[HDFS-16566] Erasure Coding: Recovery may cause excess replicas when busy DN exsits

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.2
• Fix Version/s: 3.4.0, 3.3.6
• Component/s: ec, erasure-coding
• Labels: pull-request-available
• Assignee: Ruinan Gu Assign to me
• Reporter: Ruinan Gu
• Created: 29/Apr/22 10:15
• Updated: 08/Jun/23 20:26
• Resolved: 15/Jul/22 20:52

Description

Simple case:

RS3-2 ,[0(busy),2,3,4] (1 missing),0 is busy.

We can get liveblockIndice=[2,3,4], additionalRepl=1.So the DN will get the LiveBitSet=[2,3,4] and targets.length=1.

According to StripedWriter.initTargetIndices(), 0 will get recovered instead of 1. So the internal blocks will become [0(busy),2,3,4,0'(excess)].Although NN will detect, delete the excess replicas and recover the missing block(1) correctly after the wrong recovery of 0', I don't think this process is expected and the recovery of 0' is obviously wrong and not necessary.

Issue Links

• is related to: HDFS-16874 Improve DataNode decommission for Erasure Coding
• links to: GitHub Pull Request #4252
• links to: GitHub Pull Request #5059