Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-16829 Über-jira: S3A Hadoop 3.4 features
  3. HADOOP-16756

distcp -update to S3A; abfs, etc always overwrites due to block size mismatch

    XMLWordPrintableJSON

    Details

      Description

      Distcp over S3A always copies all source files no matter the files are changed or not. This is opposite to the statement in the doc below.

      http://hadoop.apache.org/docs/current/hadoop-distcp/DistCp.html

      And to use -update to only copy changed files.
      

      CopyMapper compares file length as well as block size before copying. While the file length should match, the block size does not. This is apparently because the returned block size from S3A is always 32MB.

      https://github.com/apache/hadoop/blob/release-3.2.0-RC1/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java#L348

      I'd suppose we should update the documentation or make code change.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                stevel@apache.org Steve Loughran
                Reporter:
                daisuke.kobayashi Daisuke Kobayashi
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: