[HDFS-8147] Mover should not schedule two replicas to the same DN storage

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.6.0
• Fix Version/s: 2.8.0, 2.7.1, 3.0.0-alpha1
• Component/s: balancer & mover
• Hadoop Flags: Reviewed
• Assignee: Surendra Singh Lilhore
• Reporter: Surendra Singh Lilhore
• Created: 15/Apr/15 05:17
• Updated: 06/Jan/17 01:33
• Resolved: 23/Apr/15 18:59

Description

Scenario:
1. Three DN cluster. For DNs storage type is like this.
DN1 : DISK,ARCHIVE
DN2 : DISK
DN3 : DISK,ARCHIVE (All DNs are in same rack)
2. One file with two replicas (In DN1 and DN2)
3. Set file storage policy COLD
4. Now execute Mover.

Expected Result: File blocks should move in DN1:ARCHIVE and DN3:ARCHIVE

Actual Result: chooseTargetInSameNode() move D1:DISK block to D1:ARCHIVE, but in next iteration chooseTarget() for same rake is selecting again DN1:ARCHIVE for target where already same block exists.
chooseTargetInSameNode() and chooseTarget() should not select the node as target where already same replica exists. The dispatcher will fail to move block as shown in the log below. Then, the Mover will try again in next iteration.

Logs

15/04/15 10:47:17 WARN balancer.Dispatcher: Failed to move blk_1073741852_1028 with size=11990 from 10.19.92.74:50010:DISK to 10.19.92.73:50010:ARCHIVE through 10.19.92.73:50010: Got error, status message opReplaceBlock BP-1258709199-10.19.92.74-1428292615636:blk_1073741852_1028 received exception org.apache.hadoop.hdfs.server.datanode.ReplicaAlreadyExistsException: Replica FinalizedReplica, blk_1073741852_1028, FINALIZED

Issue Links

• is broken by: HDFS-7310 Mover can give first priority to local DN if it has target storage type available in local DN
• is related to: HDFS-8143 HDFS Mover tool should exit after some retry when failed to move blocks.
• is related to: HDFS-8204 Mover/Balancer should not schedule two replicas to the same DN
• is required by: HDFS-7621 Erasure Coding: update the Balancer/Mover data migration logic