[HDFS-3318] Hftp hangs on transfers >2GB

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.3, 2.0.0-alpha
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: hdfs-client
• Target Version/s: 0.23.3
• Hadoop Flags: Incompatible change, Reviewed
• Assignee: Daryn Sharp
• Reporter: Daryn Sharp
• Created: 24/Apr/12 21:52
• Updated: 10/Mar/15 04:35
• Resolved: 25/Apr/12 19:26

Description

Hftp transfers >2GB hang after the transfer is complete. The problem appears to be caused by java internally using an int for the content length. When it overflows 2GB, it won't check the bounds of the reads on the input stream. The client continues reading after all data is received, and the client blocks until the server times out the connection – many minutes later. In conjunction with hftp timeouts, all transfers >2G fail with a read timeout.

Issue Links

• breaks: HDFS-3577 WebHdfsFileSystem can not read files larger than 24KB
• is related to: HDFS-4715 Backport HDFS-3577 and other related WebHDFS issue to branch-1
• is related to: HDFS-3671 ByteRangeInputStream shouldn't require the content length header be present
• relates to: HDFS-3166 Hftp connections do not have a timeout
• relates to: HDFS-1085 hftp read failing silently