[HDFS-10530] BlockManager reconstruction work scheduling should correctly adhere to EC block placement policy

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-alpha4
• Component/s: namenode
• Labels: hdfs-ec-3.0-nice-to-have
• Assignee: Manoj Govindassamy Assign to me
• Reporter: Rui Gao
• Created: 15/Jun/16 05:40
• Updated: 22/Mar/17 01:03
• Resolved: 16/Mar/17 22:08

Description

This issue was found by Takuya Fukudome.

Under RS-DEFAULT-6-3-64k EC policy,
1. Create an EC file, the file was witten to all the 5 racks( 2 dns for each) of the cluster.
2. Reconstruction work would be scheduled if the 6th rack is added.
3. While adding the 7th rack or more racks will not trigger reconstruction work.

Based on default EC block placement policy defined in “BlockPlacementPolicyRackFaultTolerant.java”, EC file should be able to be scheduled to distribute to 9 racks if possible.

In BlockManager#isPlacementPolicySatisfied(BlockInfo storedBlock) , numReplicas of striped blocks might should be getRealTotalBlockNum(), instead of getRealDataBlockNum().