Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-8703

distcpV2: turn CRC checking off for 0 byte size

Log workAgile BoardRank to TopRank to BottomAttach filesAttach ScreenshotBulk Copy AttachmentsBulk Move AttachmentsVotersWatch issueWatchersCreate sub-taskConvert to sub-taskMoveLinkCloneLabelsUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Fixed
    • 0.23.3
    • 0.23.3, 2.0.2-alpha
    • None
    • None
    • distcp skips CRC on 0 byte files.

    Description

      DistcpV2 (hadoop-tools/hadoop-distcp/..) can fail from checksum failure, sometimes when copying a 0 byte file. Root cause of this may have to do with an inconsistent nature of HDFS when creating 0 byte files, however distcp can avoid this issue by not checking CRC when size is zero.

      This issue was reported as part of HADOOP-8233, though it seems like a better idea to treat this particular aspect on it's own.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            davet Dave Thompson Assign to me
            davet Dave Thompson
            Votes:
            0 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment