[HDFS-10178] Permanent write failures can happen if pipeline recoveries occur for the first packet

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 2.8.0, 2.7.3, 2.6.5, 3.0.0-alpha1
• Target Version/s: 2.7.3, 2.6.5
• Hadoop Flags: Reviewed
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 17/Mar/16 21:48
• Updated: 06/Jan/17 00:48
• Resolved: 04/Apr/16 21:57

Description

We have observed that write fails permanently if the first packet doesn't go through properly and pipeline recovery happens. If the write op creates a pipeline, but the actual data packet does not reach one or more datanodes in time, the pipeline recovery will be done against the 0-byte partial block.

If additional datanodes are added, the block is transferred to the new nodes. After the transfer, each node will have a meta file containing the header and 0-length data block file. The pipeline recovery seems to work correctly up to this point, but write fails when actual data packet is resent.

Issue Links

• is related to: HDFS-10490 Client may never recovery replica after a timeout during sending packet