Hadoop Common
  1. Hadoop Common
  2. HADOOP-1652

Rebalance data blocks when new data nodes added or data nodes become full

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.13.0
    • Fix Version/s: 0.16.0
    • Component/s: None
    • Labels:
      None

      Description

      When a new data node joins hdfs cluster, it does not hold much data. So any map task assigned to the machine most likely does not read local data, thus increasing the use of network bandwidth. On the other hand, when some data nodes become full, new data blocks are placed on only non-full data nodes, thus reducing their read parallelism.

      This jira aims to find an approach to redistribute data blocks when imbalance occurs in the cluster. An solution should meet the following requirements:
      1. It maintains data availablility guranteens in the sense that rebalancing does not reduce the number of replicas that a block has or the number of racks that the block resides.
      2. An adminstrator should be able to invoke and interrupt rebalancing from a command line.
      3. Rebalancing should be throttled so that rebalancing does not cause a namenode to be too busy to serve any incoming request or saturate the network.

      1. RebalanceDesign4.pdf
        47 kB
        Hairong Kuang
      2. RebalanceDesign5.pdf
        45 kB
        Hairong Kuang
      3. RebalanceDesign6.pdf
        50 kB
        Hairong Kuang
      4. balancer.patch
        51 kB
        Hairong Kuang
      5. BalancerAdminGuide.pdf
        13 kB
        Hairong Kuang
      6. balancer1.patch
        65 kB
        Hairong Kuang
      7. balancer2.patch
        71 kB
        Hairong Kuang
      8. BalancerAdminGuide1.pdf
        14 kB
        Hairong Kuang
      9. balancer3.patch
        75 kB
        Hairong Kuang
      10. balancer4.patch
        71 kB
        Hairong Kuang
      11. balancer5.patch
        71 kB
        Hairong Kuang
      12. BalancerUserGuide2.pdf
        14 kB
        Hairong Kuang
      13. balancer6.patch
        71 kB
        Hairong Kuang
      14. balancer7.patch
        72 kB
        Hairong Kuang
      15. balancer8.patch
        72 kB
        Hairong Kuang
      16. Balancer.html
        20 kB
        Ravi Phulari

        Issue Links

          Activity

          No work has yet been logged on this issue.

            People

            • Assignee:
              Hairong Kuang
              Reporter:
              Hairong Kuang
            • Votes:
              1 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development