Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13377 AliyunOSS: improvements for stabilization and optimization
  3. HADOOP-15323

AliyunOSS: Improve copy file performance for AliyunOSSFileSystemStore

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.10.0, 3.2.0, 3.1.1, 2.9.2, 3.0.3, 3.3.0
    • Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 3.1.2, 3.2.1, 2.9.3
    • Component/s: fs/oss
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Aliyun OSS will support shallow copy which means server will only copy metadata when copy object operation occurs. 

      With shallow copy, we can use copyObject api instead of multi-part copy api if we do not change object storage type & encryption type & source object  is uploaded by Put / Multipart upload api.

      We will try to use copyObject api, and check result. If shallow copy disabled for this object, then we will use multipart copy. So, I will remove fs.oss.multipart.upload.threshold configuration.

        Attachments

        1. HADOOP-15323.001.patch
          12 kB
          wujinhu
        2. HADOOP-15323.002.patch
          15 kB
          wujinhu
        3. HADOOP-15323.003.patch
          14 kB
          wujinhu

          Activity

            People

            • Assignee:
              wujinhu wujinhu
              Reporter:
              wujinhu wujinhu
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: