XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.8.0
    • 3.0.0-beta1
    • fs/s3
    • None
    • S3A now defaults to using the "v2" S3 list API, which speeds up large-scale path listings. Non-AWS S3 implementations may not support this API: consult the S3A documentation on how to revert to the v1 API.

    Description

      Unlike version 1 of the S3 List Objects API, version 2 by default does not fetch object owner information, which S3A doesn't need anyway. By switching to v2, there will be less data to transfer/process. Also, it should be more robust when listing a versioned bucket with "a large number of delete markers" (according to AWS).

      Methods in S3AFileSystem that use this API include:

      • getFileStatus(Path)
      • innerDelete(Path, boolean)
      • innerListStatus(Path)
      • innerRename(Path, Path)

      Requires AWS SDK 1.10.75 or later.

      Attachments

        1. HADOOP-13421.004.patch
          37 kB
          Aaron Fabbri
        2. HADOOP-13421.003.patch
          36 kB
          Aaron Fabbri
        3. HADOOP-13421.002.patch
          35 kB
          Aaron Fabbri
        4. HADOOP-13421-HADOOP-13345.001.patch
          27 kB
          Aaron Fabbri

        Issue Links

          Activity

            People

              fabbri Aaron Fabbri
              slider Steven K. Wong
              Votes:
              3 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: