[HDFS-9818] Correctly handle EC reconstruction work caused by not enough racks

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha1
• Fix Version/s: 3.0.0-alpha1
• Component/s: datanode, namenode
• Hadoop Flags: Reviewed
• Assignee: Jing Zhao
• Reporter: Takuya Fukudome
• Created: 17/Feb/16 01:22
• Updated: 12/May/16 18:12
• Resolved: 20/Feb/16 03:03

Description

This is reported by tfukudom:

In a system test where 1 of 7 datanode racks were stopped, HadoopIllegalArgumentException was seen on DataNode side while reconstructing missing EC blocks:

2016-02-16 11:09:06,672 WARN  datanode.DataNode (ErasureCodingWorker.java:run(482)) - Failed to recover striped block: BP-480558282-172.29.4.13-1453805190696:blk_-9223372036850962784_278270
org.apache.hadoop.HadoopIllegalArgumentException: Inputs not fully corresponding to erasedIndexes in null places. erasedOrNotToReadIndexes: [1, 2, 6], erasedIndexes: [3]
	at org.apache.hadoop.io.erasurecode.rawcoder.RSRawDecoder.doDecode(RSRawDecoder.java:166)
	at org.apache.hadoop.io.erasurecode.rawcoder.AbstractRawErasureDecoder.decode(AbstractRawErasureDecoder.java:84)
	at org.apache.hadoop.io.erasurecode.rawcoder.RSRawDecoder.decode(RSRawDecoder.java:89)
	at org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$ReconstructAndTransferBlock.recoverTargets(ErasureCodingWorker.java:683)
	at org.apache.hadoop.hdfs.server.datanode.erasurecode.ErasureCodingWorker$ReconstructAndTransferBlock.run(ErasureCodingWorker.java:465)

Issue Links

• relates to: HDFS-8786 Erasure coding: use simple replication for internal blocks on decommissioning datanodes