[HADOOP-8233] Turn CRC checking off for 0 byte size and differing blocksizes

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.23.3
• Component/s: tools/distcp
• Assignee: Dave Thompson
• Reporter: Dave Thompson
• Created: 30/Mar/12 22:14
• Updated: 21/Jan/21 13:18

Description

DistcpV2 (hadoop-tools/hadoop-distcp/..) can fail from checksum failure, sometimes when copying a 0 byte file. Root cause of this may have to do with an inconsistent nature of HDFS when creating 0 byte files, however distcp can avoid this issue by not checking CRC when size is zero.

Further, distcp fails checksum when copying from two clusters that use different blocksizes. In this case it does not make sense to check CRC, as it is a guaranteed failure.

We need to turn CRC checking off for the above two cases.

Issue Links

• is depended upon by: HADOOP-15788 Improve Distcp for long-haul/cloud deployments