[HDFS-16127] Improper pipeline close recovery causes a permanent write failure or data loss.

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.2
• Fix Version/s: 3.4.0, 3.2.3, 3.3.2
• Component/s: hdfs
• Target Version/s: 3.4.0, 3.2.3, 3.3.2
• Hadoop Flags: Reviewed
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 13/Jul/21 14:51
• Updated: 27/Jan/24 02:42
• Resolved: 16/Jul/21 19:26

Description

When a block is being closed, the data streamer in the client waits for the final ACK to be delivered. If an exception is received during this wait, the close is retried. This assumption has become invalid by HDFS-15813, resulting in permanent write failures in some close error cases involving slow nodes. There are also less frequent cases of data loss.

Issue Links

• is caused by: HDFS-15813 DataStreamer: keep sending heartbeat packets while streaming