[HDFS-14699] Erasure Coding: Storage not considered in live replica when replication streams hard limit reached to threshold

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.2.0, 3.1.1, 3.3.0
• Fix Version/s: 3.3.0, 3.1.4, 3.2.2
• Component/s: ec
• Labels: patch
• Assignee: Zhao Yi Ming
• Reporter: Zhao Yi Ming
• Created: 05/Aug/19 03:27
• Updated: 07/Nov/19 10:04
• Resolved: 12/Sep/19 14:06

Description

We are tried the EC function on 80 node cluster with hadoop 3.1.1, we hit the same scenario as you said https://issues.apache.org/jira/browse/HDFS-8881. Following are our testing steps, hope it can helpful.(following DNs have the testing internal blocks)

1. we customized a new 10-2-1024k policy and use it on a path, now we have 12 internal block(12 live block)
2. decommission one DN, after the decommission complete. now we have 13 internal block(12 live block and 1 decommission block)
3. then shutdown one DN which did not have the same block id as 1 decommission block, now we have 12 internal block(11 live block and 1 decommission block)
4. after wait for about 600s (before the heart beat come) commission the decommissioned DN again, now we have 12 internal block(11 live block and 1 duplicate block)
5. Then the EC is not reconstruct the missed block

We think this is a critical issue for using the EC function in a production env. Could you help? Thanks a lot!

Issue Links

• is duplicated by: HDFS-8881 Erasure Coding: internal blocks got missed and got over-replicated at the same time
• links to: GitHub Pull Request #1337