Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Minor
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 3.0.0-beta1
    • Component/s: fs/s3
    • Labels:
      None
    • Release Note:
      S3A now defaults to using the "v2" S3 list API, which speeds up large-scale path listings. Non-AWS S3 implementations may not support this API: consult the S3A documentation on how to revert to the v1 API.

      Description

      Unlike version 1 of the S3 List Objects API, version 2 by default does not fetch object owner information, which S3A doesn't need anyway. By switching to v2, there will be less data to transfer/process. Also, it should be more robust when listing a versioned bucket with "a large number of delete markers" (according to AWS).

      Methods in S3AFileSystem that use this API include:

      • getFileStatus(Path)
      • innerDelete(Path, boolean)
      • innerListStatus(Path)
      • innerRename(Path, Path)

      Requires AWS SDK 1.10.75 or later.

        Attachments

        1. HADOOP-13421-HADOOP-13345.001.patch
          27 kB
          Aaron Fabbri
        2. HADOOP-13421.002.patch
          35 kB
          Aaron Fabbri
        3. HADOOP-13421.003.patch
          36 kB
          Aaron Fabbri
        4. HADOOP-13421.004.patch
          37 kB
          Aaron Fabbri

          Issue Links

            Activity

              People

              • Assignee:
                fabbri Aaron Fabbri
                Reporter:
                slider Steven K. Wong
              • Votes:
                3 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: