Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11384

Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike

          Details

          • Target Version/s:
          • Hadoop Flags:
            Reviewed

            Description

            When running balancer on hadoop cluster which have more than 3000 Datanodes will cause NameNode's rpc.CallQueueLength spike. We observed this situation could cause Hbase cluster failure due to RegionServer's WAL timeout.

              Attachments

              1. HDFS-11384-branch-2.7.011.patch
                13 kB
                Konstantin Shvachko
              2. HDFS-11384-branch-2.8.011.patch
                13 kB
                Konstantin Shvachko
              3. HDFS-11384.011.patch
                16 kB
                Konstantin Shvachko
              4. HDFS-11384.010.patch
                16 kB
                Konstantin Shvachko
              5. HDFS-11384.009.patch
                14 kB
                Konstantin Shvachko
              6. HDFS-11384.008.patch
                15 kB
                Konstantin Shvachko
              7. HDFS-11384-007.patch
                13 kB
                Konstantin Shvachko
              8. HDFS-11384.006.patch
                13 kB
                Konstantin Shvachko
              9. HDFS-11384.005.patch
                5 kB
                Konstantin Shvachko
              10. HDFS-11384.004.patch
                5 kB
                Konstantin Shvachko
              11. HDFS-11384.003.patch
                5 kB
                Konstantin Shvachko
              12. HDFS-11384.002.patch
                5 kB
                yunjiong zhao
              13. HDFS-11384.001.patch
                4 kB
                yunjiong zhao
              14. balancer.day.png
                58 kB
                yunjiong zhao
              15. balancer.week.png
                50 kB
                yunjiong zhao

                Issue Links

                breaks

                Bug - A problem which impairs or prevents the functions of the product. HDFS-12347 TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently

                • Critical - Crashes, loss of data, severe memory leak.
                • Resolved
                is related to

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-8825 Enhancements to Balancer

                • Major - Major loss of function.
                • Open

                  Activity

                    People

                    • Assignee:
                      shv Konstantin Shvachko
                      Reporter:
                      zhaoyunjiong yunjiong zhao
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      11 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: