Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8233

Turn CRC checking off for 0 byte size and differing blocksizes

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 0.23.3
    • Fix Version/s: None
    • Component/s: tools/distcp
    • Labels:
      None

      Description

      DistcpV2 (hadoop-tools/hadoop-distcp/..) can fail from checksum failure, sometimes when copying a 0 byte file. Root cause of this may have to do with an inconsistent nature of HDFS when creating 0 byte files, however distcp can avoid this issue by not checking CRC when size is zero.

      Further, distcp fails checksum when copying from two clusters that use different blocksizes. In this case it does not make sense to check CRC, as it is a guaranteed failure.

      We need to turn CRC checking off for the above two cases.

        Attachments

        1. HADOOP-8233-branch-0.23.2.patch
          2 kB
          Dave Thompson

          Issue Links

            Activity

              People

              • Assignee:
                davet Dave Thompson
                Reporter:
                davet Dave Thompson
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated: