[HDFS-14581] Appending to EC files crashes NameNode

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: erasure-coding
• Assignee: Surendra Singh Lilhore
• Reporter: Surendra Singh Lilhore
• Created: 18/Jun/19 11:41
• Updated: 02/Oct/19 17:15
• Resolved: 20/Jun/19 16:41

Description

org.apache.hadoop.hdfs.server.namenode.FSDirAppendOp.prepareFileForAppend(..)@189

file.recordModification(iip.getLatestSnapshotId());
file.toUnderConstruction(leaseHolder, clientMachine);

fsn.getLeaseManager().addLease(
file.getFileUnderConstructionFeature().getClientName(), file.getId());

LocatedBlock ret = null;
if (!newBlock) {
if (file.isStriped()) {
throw new UnsupportedOperationException(
"Append on EC file without new block is not supported.");
}

In this code "UnsupportedOperationException" exception thows after marking file underConstruction. In this case file is opened without any "Open" editlogs, after some time lease manager close this file and add close edit log.

When SBN tail this edit log, it will fail with this exception.

2019-06-13 19:17:51,513 ERROR org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader: Encountered exception on operation CloseOp [length=0, inodeId=0, path=/ECtest/********, replication=1, mtime=1560261947480, atime=1560258249117, blockSize=134217728, blocks=[blk_-9223372036854775792_1005], permissions=root:hadoop:rw-r--r--, aclEntries=null, clientName=, clientMachine=, overwrite=false, storagePolicyId=0, erasureCodingPolicyId=0, opCode=OP_CLOSE, txid=1363]
java.io.IOException: File is not under construction: /ECtest/container-executor
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.applyEditLogOp(FSEditLogLoader.java:504)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadEditRecords(FSEditLogLoader.java:286)
at org.apache.hadoop.hdfs.server.namenode.FSEditLogLoader.loadFSEdits(FSEditLogLoader.java:181)
at org.apache.hadoop.hdfs.server.namenode.FSImage.loadEdits(FSImage.java:924)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:329)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:410)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
at org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:485)
at org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)

Issue Links

• is blocked by: HDFS-7663 Erasure Coding: Append on striped file