Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-17566 Über-jira: S3A Hadoop 3.3.2 features
  3. HADOOP-17139

Re-enable optimized copyFromLocal implementation in S3AFileSystem

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.3.0, 3.2.1
    • 3.3.2
    • fs/s3

    Description

      It looks like HADOOP-15932 disabled the optimized copyFromLocal implementation in S3A for correctness reasons.  innerCopyFromLocalFile should be fixed and re-enabled. The current implementation uses FileSystem.copyFromLocal which will open an input stream from the local fs and an output stream to the destination fs, and then call IOUtils.copyBytes. With default configs, this will cause S3A to read the file into memory, write it back to a file on the local fs, and then when the file is closed, upload it to S3.

      The optimized version of copyFromLocal in innerCopyFromLocalFile, directly creates a PutObjectRequest request with the local file as the input.

      Attachments

        Issue Links

          Activity

            People

              bogthe Bogdan Stolojan
              stakiar Sahil Takiar
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 9h 50m
                  9h 50m