[HADOOP-8060] Add a capability to discover and set checksum types per file.

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.23.0, 0.23.1, 0.24.0
• Fix Version/s: 0.23.3, 2.0.2-alpha
• Component/s: fs, util
• Target Version/s: 0.23.3
• Assignee: Kihwal Lee
• Reporter: Kihwal Lee
• Created: 11/Feb/12 23:59
• Updated: 11/Oct/12 17:45
• Resolved: 27/Aug/12 16:46

Description

After the improved CRC32C checksum feature became default, some of use cases involving data movement are no longer supported. For example, when running DistCp to copy from a file stored with the CRC32 checksum to a new cluster with the CRC32C set to default checksum, the final data integrity check fails because of mismatch in checksums.

Issue Links

• incorporates: HDFS-3176 JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own.
• incorporates: HDFS-3177 Allow DFSClient to find out and use the CRC type being used for a file.
• incorporates: HADOOP-8239 Extend MD5MD5CRC32FileChecksum to show the actual checksum type being used
• incorporates: HADOOP-8240 Allow users to specify a checksum type on create()
• relates to: HDFS-3054 distcp -skipcrccheck has no effect