[HDFS-7587] Edit log corruption can happen if append fails with a quota violation

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 2.7.0, 2.6.1, 3.0.0-alpha1
• Component/s: namenode
• Labels: 2.6.1-candidate
• Target Version/s: 2.7.0
• Hadoop Flags: Reviewed
• Assignee: Jing Zhao
• Reporter: Kihwal Lee
• Created: 06/Jan/15 20:11
• Updated: 30/Aug/16 01:40
• Resolved: 19/Mar/15 02:02

Description

We have seen a standby namenode crashing due to edit log corruption. It was complaining that OP_CLOSE cannot be applied because the file is not under-construction.

When a client was trying to append to the file, the remaining space quota was very small. This caused a failure in prepareFileForWrite(), but after the inode was already converted for writing and a lease added. Since these were not undone when the quota violation was detected, the file was left in under-construction with an active lease without edit logging OP_ADD.

A subsequent append() eventually caused a lease recovery after the soft limit period. This resulted in commitBlockSynchronization(), which closed the file with OP_CLOSE being logged. Since there was no corresponding OP_ADD, edit replaying could not apply this.

Issue Links

• is broken by: HDFS-6423 Diskspace quota usage should be updated when appending data to partial block
• relates to: HDFS-8531 Append failed due to unreleased lease from previous appender with quota exceeded exception