Issue Details (XML | Word | Printable)

Key: HADOOP-1652
Type: New Feature New Feature
Status: Closed Closed
Resolution: Fixed
Priority: Major Major
Assignee: Hairong Kuang
Reporter: Hairong Kuang
Votes: 1
Watchers: 0
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

Rebalance data blocks when new data nodes added or data nodes become full

Created: 25/Jul/07 06:47 PM   Updated: 13/Oct/09 09:31 PM
Return to search
Component/s: None
Affects Version/s: 0.13.0
Fix Version/s: 0.16.0

Time Tracking:
Not Specified

File Attachments:
  Size
HTML File Licensed for inclusion in ASF works Balancer.html 2009-10-13 09:31 PM Ravi Phulari 20 kB
Text File Licensed for inclusion in ASF works balancer.patch 2007-10-26 08:51 PM Hairong Kuang 51 kB
Text File Licensed for inclusion in ASF works balancer1.patch 2007-11-09 08:12 PM Hairong Kuang 65 kB
Text File Licensed for inclusion in ASF works balancer2.patch 2007-11-20 08:38 AM Hairong Kuang 71 kB
Text File Licensed for inclusion in ASF works balancer3.patch 2007-11-26 09:18 PM Hairong Kuang 75 kB
Text File Licensed for inclusion in ASF works balancer4.patch 2007-11-29 06:17 PM Hairong Kuang 71 kB
Text File Licensed for inclusion in ASF works balancer5.patch 2007-12-04 06:50 PM Hairong Kuang 71 kB
Text File Licensed for inclusion in ASF works balancer6.patch 2007-12-04 10:10 PM Hairong Kuang 71 kB
Text File Licensed for inclusion in ASF works balancer7.patch 2007-12-05 12:52 AM Hairong Kuang 72 kB
Text File Licensed for inclusion in ASF works balancer8.patch 2007-12-05 06:46 PM Hairong Kuang 72 kB
PDF File Licensed for inclusion in ASF works BalancerAdminGuide.pdf 2007-11-08 11:26 PM Hairong Kuang 13 kB
PDF File Licensed for inclusion in ASF works BalancerAdminGuide1.pdf 2007-11-20 08:43 AM Hairong Kuang 14 kB
PDF File Licensed for inclusion in ASF works BalancerUserGuide2.pdf 2007-12-04 07:48 PM Hairong Kuang 14 kB
PDF File Licensed for inclusion in ASF works RebalanceDesign4.pdf 2007-08-10 10:55 PM Hairong Kuang 47 kB
PDF File Licensed for inclusion in ASF works RebalanceDesign5.pdf 2007-08-22 09:08 PM Hairong Kuang 45 kB
PDF File Licensed for inclusion in ASF works RebalanceDesign6.pdf 2007-10-23 10:48 PM Hairong Kuang 50 kB
Issue Links:
Dependants
dependent
 

Resolution Date: 05/Dec/07 07:49 PM


 Description  « Hide
When a new data node joins hdfs cluster, it does not hold much data. So any map task assigned to the machine most likely does not read local data, thus increasing the use of network bandwidth. On the other hand, when some data nodes become full, new data blocks are placed on only non-full data nodes, thus reducing their read parallelism.

This jira aims to find an approach to redistribute data blocks when imbalance occurs in the cluster. An solution should meet the following requirements:
1. It maintains data availablility guranteens in the sense that rebalancing does not reduce the number of replicas that a block has or the number of racks that the block resides.
2. An adminstrator should be able to invoke and interrupt rebalancing from a command line.
3. Rebalancing should be throttled so that rebalancing does not cause a namenode to be too busy to serve any incoming request or saturate the network.



 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Hairong Kuang made changes - 10/Aug/07 10:55 PM
Field Original Value New Value
Attachment RebalanceDesign4.pdf [ 12363636 ]
Hairong Kuang made changes - 22/Aug/07 09:08 PM
Attachment RebalanceDesign5.pdf [ 12364355 ]
Hairong Kuang made changes - 06/Sep/07 11:24 PM
Link This issue depends upon HADOOP-1266 [ HADOOP-1266 ]
Hairong Kuang made changes - 06/Sep/07 11:29 PM
Link This issue blocks HADOOP-1846 [ HADOOP-1846 ]
Hairong Kuang made changes - 06/Sep/07 11:30 PM
Link This issue blocks HADOOP-1846 [ HADOOP-1846 ]
Hairong Kuang made changes - 06/Sep/07 11:31 PM
Link This issue depends upon HADOOP-1846 [ HADOOP-1846 ]
Hairong Kuang made changes - 17/Sep/07 10:19 PM
Link This issue blocks HADOOP-1912 [ HADOOP-1912 ]
Hairong Kuang made changes - 17/Sep/07 10:22 PM
Link This issue blocks HADOOP-1912 [ HADOOP-1912 ]
Hairong Kuang made changes - 17/Sep/07 10:22 PM
Link This issue depends on HADOOP-1912 [ HADOOP-1912 ]
Hairong Kuang made changes - 18/Sep/07 12:05 AM
Link This issue depends on HADOOP-1914 [ HADOOP-1914 ]
Hairong Kuang made changes - 20/Sep/07 10:21 PM
Link This issue incorporates HADOOP-1914 [ HADOOP-1914 ]
Hairong Kuang made changes - 20/Sep/07 10:21 PM
Link This issue depends on HADOOP-1914 [ HADOOP-1914 ]
Hairong Kuang made changes - 20/Sep/07 10:22 PM
Link This issue incorporates HADOOP-1912 [ HADOOP-1912 ]
Hairong Kuang made changes - 20/Sep/07 10:23 PM
Link This issue depends on HADOOP-1912 [ HADOOP-1912 ]
Hairong Kuang made changes - 20/Sep/07 10:34 PM
Link This issue depends on HADOOP-1912 [ HADOOP-1912 ]
Hairong Kuang made changes - 20/Sep/07 10:35 PM
Link This issue depends on HADOOP-1914 [ HADOOP-1914 ]
Hairong Kuang made changes - 20/Sep/07 10:35 PM
Link This issue incorporates HADOOP-1912 [ HADOOP-1912 ]
Hairong Kuang made changes - 20/Sep/07 10:35 PM
Link This issue incorporates HADOOP-1914 [ HADOOP-1914 ]
Owen O'Malley made changes - 28/Sep/07 04:17 PM
Fix Version/s 0.15.0 [ 12312565 ]
Fix Version/s 0.16.0 [ 12312740 ]
Hairong Kuang made changes - 23/Oct/07 10:48 PM
Attachment RebalanceDesign6.pdf [ 12368261 ]
Hairong Kuang made changes - 26/Oct/07 08:51 PM
Attachment balancer.patch [ 12368519 ]
Hairong Kuang made changes - 08/Nov/07 11:26 PM
Attachment BalancerAdminGuide.pdf [ 12369201 ]
Hairong Kuang made changes - 09/Nov/07 07:10 PM
Attachment balancer1.patch [ 12369254 ]
Hairong Kuang made changes - 09/Nov/07 08:11 PM
Attachment balancer1.patch [ 12369254 ]
Hairong Kuang made changes - 09/Nov/07 08:12 PM
Attachment balancer1.patch [ 12369265 ]
Hairong Kuang made changes - 20/Nov/07 08:38 AM
Attachment balancer2.patch [ 12369856 ]
Hairong Kuang made changes - 20/Nov/07 08:43 AM
Attachment BalancerAdminGuide1.pdf [ 12369857 ]
Hairong Kuang made changes - 26/Nov/07 09:18 PM
Attachment balancer3.patch [ 12370239 ]
Hairong Kuang made changes - 29/Nov/07 06:17 PM
Attachment balancer4.patch [ 12370565 ]
Hairong Kuang made changes - 04/Dec/07 06:50 PM
Attachment balancer5.patch [ 12370962 ]
Hairong Kuang made changes - 04/Dec/07 06:50 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Hairong Kuang made changes - 04/Dec/07 07:48 PM
Attachment BalancerUserGuide2.pdf [ 12370966 ]
Hairong Kuang made changes - 04/Dec/07 10:09 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Hairong Kuang made changes - 04/Dec/07 10:10 PM
Attachment balancer6.patch [ 12370975 ]
Hairong Kuang made changes - 04/Dec/07 10:10 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Hairong Kuang made changes - 05/Dec/07 12:46 AM
Status Patch Available [ 10002 ] Open [ 1 ]
Hairong Kuang made changes - 05/Dec/07 12:52 AM
Attachment balancer7.patch [ 12370987 ]
Hairong Kuang made changes - 05/Dec/07 12:52 AM
Status Open [ 1 ] Patch Available [ 10002 ]
Hairong Kuang made changes - 05/Dec/07 06:46 PM
Attachment balancer8.patch [ 12371057 ]
dhruba borthakur made changes - 05/Dec/07 07:49 PM
Resolution Fixed [ 1 ]
Status Patch Available [ 10002 ] Resolved [ 5 ]
Jim Kellerman made changes - 15/Jan/08 07:42 PM
Link This issue blocks HADOOP-2138 [ HADOOP-2138 ]
Nigel Daley made changes - 08/Feb/08 11:37 PM
Status Resolved [ 5 ] Closed [ 6 ]
Owen O'Malley made changes - 08/Jul/09 04:42 PM
Component/s dfs [ 12310710 ]
Ravi Phulari made changes - 13/Oct/09 09:31 PM
Attachment Balancer.html [ 12422024 ]