Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11384

Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike

VotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      When running balancer on hadoop cluster which have more than 3000 Datanodes will cause NameNode's rpc.CallQueueLength spike. We observed this situation could cause Hbase cluster failure due to RegionServer's WAL timeout.

        Attachments

        1. HDFS-11384-branch-2.8.011.patch
          13 kB
          Konstantin Shvachko
        2. HDFS-11384-branch-2.7.011.patch
          13 kB
          Konstantin Shvachko
        3. HDFS-11384-007.patch
          13 kB
          Konstantin Shvachko
        4. HDFS-11384.011.patch
          16 kB
          Konstantin Shvachko
        5. HDFS-11384.010.patch
          16 kB
          Konstantin Shvachko
        6. HDFS-11384.009.patch
          14 kB
          Konstantin Shvachko
        7. HDFS-11384.008.patch
          15 kB
          Konstantin Shvachko
        8. HDFS-11384.006.patch
          13 kB
          Konstantin Shvachko
        9. HDFS-11384.005.patch
          5 kB
          Konstantin Shvachko
        10. HDFS-11384.004.patch
          5 kB
          Konstantin Shvachko
        11. HDFS-11384.003.patch
          5 kB
          Konstantin Shvachko
        12. HDFS-11384.002.patch
          5 kB
          yunjiong zhao
        13. HDFS-11384.001.patch
          4 kB
          yunjiong zhao
        14. balancer.week.png
          50 kB
          yunjiong zhao
        15. balancer.day.png
          58 kB
          yunjiong zhao

        Issue Links

          Activity

            People

            • Assignee:
              shv Konstantin Shvachko
              Reporter:
              zhaoyunjiong yunjiong zhao

              Dates

              • Created:
                Updated:
                Resolved:

                Issue deployment