Details
-
Bug
-
Status: Closed
-
Blocker
-
Resolution: Fixed
-
0.23.3, 2.0.0-alpha
-
None
-
Reviewed
Description
If reading a file large enough for which the httpserver running webhdfs/httpfs uses chunked transfer encoding (more than 24K in the case of webhdfs), then the WebHdfsFileSystem client fails with an IOException with message Content-Length header is missing.
It looks like WebHdfsFileSystem is delegating opening of the inputstream to ByteRangeInputStream.URLOpener class, which checks for the Content-Length header, but when using chunked transfer encoding the Content-Length header is not present and the URLOpener.openInputStream() method thrown an exception.