[HDFS-3696] Create files with WebHdfsFileSystem goes OOM when file size is big

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.0.0-alpha
• Fix Version/s: 1.1.0, 0.23.3, 2.0.2-alpha
• Component/s: webhdfs
• Target Version/s: 1.1.0
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Kihwal Lee
• Created: 20/Jul/12 14:23
• Updated: 27/Mar/13 10:40
• Resolved: 27/Jul/12 06:34

Description

When doing "fs -put" to a WebHdfsFileSystem (webhdfs://), the FsShell goes OOM if the file size is large. When I tested, 20MB files were fine, but 200MB didn't work.

I also tried reading a large file by issuing "-cat" and piping to a slow sink in order to force buffering. The read path didn't have this problem. The memory consumption stayed the same regardless of progress.

Issue Links

• relates to: HDFS-3671 ByteRangeInputStream shouldn't require the content length header be present