Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-13345 S3Guard: Improved Consistency for S3A
  3. HADOOP-14645

add a predicate/option to probe an S3A FS for being consistent

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: HADOOP-13345
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      The committer work shows It's useful for s3a-aware code to be able to ask if the FS is consistent fir listings. AWS S3 is when s3guard is turned on, but third-party S3 implementations may also be consistent.

      The committer should be able to ask an FS if it is consistent, something which can be

      • off by default against AWS
      • configured to be one for all buckets (external s3 server)
      • dynamically set when s3guard is enabled.

      This could be done with some predicate isListingConsistent(), or just probed as an FS option (fs.s3a.consistent.listing) which could be set explicitly in configs, and set dynamically when s3guard is on. That's actually the simpleset route all round

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              stevel@apache.org Steve Loughran
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: