[HDFS-3577] WebHdfsFileSystem can not read files larger than 24KB

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.23.3, 2.0.0-alpha
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: webhdfs
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Alejandro Abdelnur
• Created: 28/Jun/12 21:18
• Updated: 18/Apr/13 21:29
• Resolved: 20/Jul/12 00:37

Description

If reading a file large enough for which the httpserver running webhdfs/httpfs uses chunked transfer encoding (more than 24K in the case of webhdfs), then the WebHdfsFileSystem client fails with an IOException with message Content-Length header is missing.

It looks like WebHdfsFileSystem is delegating opening of the inputstream to ByteRangeInputStream.URLOpener class, which checks for the Content-Length header, but when using chunked transfer encoding the Content-Length header is not present and the URLOpener.openInputStream() method thrown an exception.

Issue Links

• is broken by: HDFS-3318 Hftp hangs on transfers >2GB
• is related to: HDFS-4715 Backport HDFS-3577 and other related WebHDFS issue to branch-1
• relates to: HDFS-3788 distcp can't copy large files using webhdfs due to missing Content-Length header
• relates to: HDFS-3671 ByteRangeInputStream shouldn't require the content length header be present