[HDFS-3788] distcp can't copy large files using webhdfs due to missing Content-Length header

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.23.3, 2.0.0-alpha
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: webhdfs
• Hadoop Flags: Reviewed
• Assignee: Tsz-wo Sze
• Reporter: Eli Collins
• Created: 12/Aug/12 21:50
• Updated: 30/Jul/13 05:29
• Resolved: 17/Aug/12 01:45

Description

The following command fails when data1 contains a 3gb file. It passes when using hftp or when the directory just contains smaller (<2gb) files, so looks like a webhdfs issue with large files.

hadoop distcp webhdfs://eli-thinkpad:50070/user/eli/data1 hdfs://localhost:8020/user/eli/data2

Issue Links

• is related to: HDFS-3577 WebHdfsFileSystem can not read files larger than 24KB
• is related to: HDFS-4715 Backport HDFS-3577 and other related WebHDFS issue to branch-1
• relates to: HDFS-3671 ByteRangeInputStream shouldn't require the content length header be present