[HDFS-15726] Client should only complete a file if the last block is finalized

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Ahmed Hussein
• Reporter: Kihwal Lee
• Created: 10/Dec/20 20:37
• Updated: 10/Dec/20 20:40

Description

We have seen certain versions of DFS client manage to call completeFile() before the blocks are finalized. This is ensured by the way colseImpl() is written. When the logic and sematics around flushing or closing changes, this can easily break, but hard to notice by just looking at the code.

We propose adding a simple yet explicit logic to prevent any violation of this protocol. completeFile() won't be called unless it has received an ack for the last packet (the one causes finalization on the datanode side). This will fix any current bug and also likely prevent inadvertently breaking it in the future.

Issue Links

• is related to: HDFS-15725 Lease Recovery never completes for a committed block which the DNs never finalize