[HDFS-14834] in some particular situation, datanode will always in DECOMMISSION_INPROGRESS state

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.2
• Component/s: hdfs

Environment

Policy：RS-6-3-1024K

Version：3.1.2

• Assignee: Unassigned
• Reporter: Janick Wu
• Created: 09/Sep/19 04:21
• Updated: 20/Jun/22 06:53
• Resolved: 20/Jun/22 06:53

Description

The file's block index is [0,1,2,3,4,5,6,7,8]. I decommission index [3] and  increase the index 5 datanode's pendingReplicationWithoutTargets.

After reconstruction of index 5， the block status is：

index	isDecommissionInProgress	state
0	false	LIVE
1	false	LIVE
2	false	LIVE
3	true	DECOMMISSIONING
4	false	LIVE
5	false	LIVE
6	false	LIVE
7	false	LIVE
8	false	LIVE
5	false	LIVE

 

In DatanodeAdminManager.Monitor thread, blockManager.countNodes(block) caculate the live bitset is {0, 1, 2, 4, 5, 6, 7, 8}, liveReplicas: 8, redundant:1

It's a low redundancy block, put it into queue and wait for schedule.

In BlockManager.RedundancyMonitor thread, live bitset is {0, 1, 2, 3, 4, 6, 7, 8} , liveReplicas:9,  redundant :0

The block waitting for replication will remove from queue, rbecause the liveReplicas satisfies the expected redundancy

 And this situation would lead to index[3]'s datanode allways in DECOMMISSION_INPROGRESS state

 

Issue Links

• is fixed by: HDFS-14754 Erasure Coding : The number of Under-Replicated Blocks never reduced