Uploaded image for project: 'Falcon'
  1. Falcon
  2. FALCON-1880

To support TDE encryption : Add --skipcrccheck to distcp options for HiveDR

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 0.9
    • 0.10
    • replication
    • None

    Description

      when using Distcp to copy the file from one Encyption to another Encryption, and it is failing with the following error:

      Caused by: java.io.IO Exception : Check-sum mismatch between hdfs://hostname:8020/user/hrt_qa/examples/input-data/text/data.txt and hdfs://hostname:8020/user/hrt_qa/examples/output-data/distcp/.distcp.tmp.attempt_1455730127943_0180_m_000000_3.
      at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.compareCheckSums(RetriableFileCopyCommand.java:210)
      at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doCopy(RetriableFileCopyCommand.java:130)
      at org.apache.hadoop.tools.mapred.RetriableFileCopyCommand.doExecute(RetriableFileCopyCommand.java:99)
      at org.apache.hadoop.tools.util.RetriableCommand.execute(RetriableCommand.java:87)
      ... 11 more
      

      By adding the parameters for Distcp "-skipcrccheck" and "-update" , the distcp passes. The following doc explains why :
      http://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/TransparentEncryption.html#a-skipcrccheck

      Attachments

        1. FALCON-1880.patch
          14 kB
          Balu Vellanki

        Activity

          People

            bvellanki Balu Vellanki
            bvellanki Balu Vellanki
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: