[FALCON-1880] To support TDE encryption : Add --skipcrccheck to distcp options for HiveDR

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9
• Fix Version/s: 0.10
• Component/s: replication
• Assignee: Balu Vellanki
• Reporter: Balu Vellanki
• Created: 31/Mar/16 22:23
• Updated: 06/Jun/16 21:25
• Resolved: 06/Apr/16 22:17

Description

when using Distcp to copy the file from one Encyption to another Encryption, and it is failing with the following error:

Caused by: java.io.IO Exception : Check-sum mismatch between hdfs://hostname:8020/user/hrt_qa/examples/input-data/text/data.txt and hdfs://hostname:8020/user/hrt_qa/examples/output-data/distcp/.distcp.tmp.attempt_1455730127943_0180_m_000000_3.
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:210)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
... 11 more

By adding the parameters for Distcp "-skipcrccheck" and "-update" , the distcp passes. The following doc explains why :
http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html#a-skipcrccheck