Uploaded image for project: 'Hadoop HDFS'
  1. Hadoop HDFS
  2. HDFS-11384

Add option for balancer to disperse getBlocks calls to avoid NameNode's rpc.CallQueueLength spike

          Details

          • Target Version/s:
          • Hadoop Flags:
            Reviewed

            Description

            When running balancer on hadoop cluster which have more than 3000 Datanodes will cause NameNode's rpc.CallQueueLength spike. We observed this situation could cause Hbase cluster failure due to RegionServer's WAL timeout.

              Attachments

              1. balancer.day.png
                58 kB
                yunjiong zhao
              2. balancer.week.png
                50 kB
                yunjiong zhao
              3. HDFS-11384.001.patch
                4 kB
                yunjiong zhao
              4. HDFS-11384.002.patch
                5 kB
                yunjiong zhao
              5. HDFS-11384.003.patch
                5 kB
                Konstantin Shvachko
              6. HDFS-11384.004.patch
                5 kB
                Konstantin Shvachko
              7. HDFS-11384.005.patch
                5 kB
                Konstantin Shvachko
              8. HDFS-11384.006.patch
                13 kB
                Konstantin Shvachko
              9. HDFS-11384.008.patch
                15 kB
                Konstantin Shvachko
              10. HDFS-11384.009.patch
                14 kB
                Konstantin Shvachko
              11. HDFS-11384.010.patch
                16 kB
                Konstantin Shvachko
              12. HDFS-11384.011.patch
                16 kB
                Konstantin Shvachko
              13. HDFS-11384-007.patch
                13 kB
                Konstantin Shvachko
              14. HDFS-11384-branch-2.7.011.patch
                13 kB
                Konstantin Shvachko
              15. HDFS-11384-branch-2.8.011.patch
                13 kB
                Konstantin Shvachko

                Issue Links

                breaks

                Bug - A problem which impairs or prevents the functions of the product. HDFS-12347 TestBalancerRPCDelay#testBalancerRPCDelay fails very frequently

                • Critical - Crashes, loss of data, severe memory leak.
                • Resolved
                is related to

                Improvement - An improvement or enhancement to an existing feature or task. HDFS-8825 Enhancements to Balancer

                • Major - Major loss of function.
                • Open

                  Activity

                    People

                    • Assignee:
                      shv Konstantin Shvachko
                      Reporter:
                      zhaoyunjiong yunjiong zhao
                    • Votes:
                      0 Vote for this issue
                      Watchers:
                      11 Start watching this issue

                      Dates

                      • Created:
                        Updated:
                        Resolved: