[HADOOP-1652] Rebalance data blocks when new data nodes added or data nodes become full

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.13.0
• Fix Version/s: 0.16.0
• Assignee: Hairong Kuang
• Reporter: Hairong Kuang
• Created: 25/Jul/07 18:47
• Updated: 25/Apr/17 20:31
• Resolved: 05/Dec/07 19:49

Description

When a new data node joins hdfs cluster, it does not hold much data. So any map task assigned to the machine most likely does not read local data, thus increasing the use of network bandwidth. On the other hand, when some data nodes become full, new data blocks are placed on only non-full data nodes, thus reducing their read parallelism.

This jira aims to find an approach to redistribute data blocks when imbalance occurs in the cluster. An solution should meet the following requirements:
1. It maintains data availablility guranteens in the sense that rebalancing does not reduce the number of replicas that a block has or the number of racks that the block resides.
2. An adminstrator should be able to invoke and interrupt rebalancing from a command line.
3. Rebalancing should be throttled so that rebalancing does not cause a namenode to be too busy to serve any incoming request or saturate the network.

Issue Links

• depends upon: HADOOP-1846 DatanodeReport should distinguish live datanodes from dead datanodes
• depends upon: HADOOP-1912 Datanode should support block replacement
• depends upon: HADOOP-1914 HDFS should have a NamenodeProtocol to allow secondary namenodes and rebalancing processes to communicate with a primary namenode
• depends upon: HADOOP-1266 Remove DatanodeDescriptor dependency from NetworkTopology
• is depended upon by: HBASE-57 [hbase] Master should allocate regions to regionservers based upon data locality and rack awareness