[HDFS-16479] EC: NameNode should not send a reconstruction work when the source datanodes are insufficient

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 3.4.0, 3.2.4, 3.3.5
• Component/s: ec, erasure-coding
• Labels: pull-request-available
• Assignee: Takanobu Asanuma
• Reporter: Yuanbo Liu
• Created: 23/Feb/22 05:57
• Updated: 14/Apr/22 03:07
• Resolved: 14/Apr/22 03:07

Description

We got this exception from DataNodes

java.lang.IllegalArgumentException: No enough live striped blocks.
        at com.google.common.base.Preconditions.checkArgument(Preconditions.java:141)
        at org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReader.<init>(StripedReader.java:128)
        at org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedReconstructor.<init>(StripedReconstructor.java:135)
        at org.apache.hadoop.hdfs.server.datanode.erasurecode.StripedBlockReconstructor.<init>(StripedBlockReconstructor.java:41)
        at org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker.processErasureCodingTasks(ErasureCodingWorker.java:133)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActive(BPOfferService.java:796)
        at org.apache.hadoop.hdfs.server.datanode.BPOfferService.processCommandFromActor(BPOfferService.java:680)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processCommand(BPServiceActor.java:1314)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.lambda$enqueue$2(BPServiceActor.java:1360)
        at org.apache.hadoop.hdfs.server.datanode.BPServiceActor$CommandProcessingThread.processQueue(BPServiceActor.java:1287)

After going through the code of ErasureCodingWork.java, we found

targets[0].getDatanodeDescriptor().addBlockToBeErasureCoded( new ExtendedBlock(blockPoolId, stripedBlk), getSrcNodes(), targets, getLiveBlockIndicies(), stripedBlk.getErasureCodingPolicy()); 

 

the liveBusyBlockIndicies is not considered as liveBlockIndicies, hence erasure coding reconstruction sometimes will fail as 'No enough live striped blocks'.

Issue Links

• relates to: HDFS-14768 EC : Busy DN replica should be consider in live replica check.
• links to: GitHub Pull Request #4138