[HDFS-1431] Balancer should work with the logic of BlockPlacementPolicy

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.22.0
• Component/s: balancer & mover
• Assignee: Scott Chen
• Reporter: Scott Chen
• Created: 30/Sep/10 00:54
• Updated: 23/Jun/15 03:59

Description

Currently Balancer does not obtain information from BlockPlacementPolicy so it can transfer the blocks without checking with BlockPlacementPolicy.
This causes the policy break after balancing the cluster.

There are some new policies proposed in HDFS-1094 and MAPREDUCE-1831 in which the block placement follows some pattern.
The pattern can be broken by Balancer.

I propose that we add the following method in BlockPlacementPolicy:

  abstract public boolean canBeMoved(String fileName, Block block,
    DatanodeInfo source, DatanodeInfo destination);

And make Balancer use it in

  private boolean isGoodBlockCandidate(Source source,
      BalancerDatanode target, BalancerBlock block)

What do you think?

Issue Links

• is related to: HDFS-7541 Upgrade Domains in HDFS