Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15220 Über-jira: S3a phase V: Hadoop 3.2 features
  3. HADOOP-15576

S3A Multipart Uploader to work with S3Guard and encryption

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 3.2.0
    • 3.2.0
    • fs/s3
    • None

    Description

      The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with the tests to demonstrate this

      1. move from low-level calls of S3A client to calls of WriteOperationHelper; adding any new methods are needed there.
      2. Tests. the tests of HDFS-13713.
      3. test execution, with -DS3Guard, -DAuth

      There isn't an S3A version of AbstractSystemMultipartUploaderTest, and even if there was, it might not show that S3Guard was bypassed, because there's no checks that listFiles/listStatus shows the newly committed files.

      Similarly, because MPU requests are initiated in S3AMultipartUploader, encryption settings are't picked up. Files being uploaded this way are not being encrypted

      Attachments

        1. HADOOP-15576.001.patch
          8 kB
          Ewan Higgs
        2. HADOOP-15576.002.patch
          24 kB
          Ewan Higgs
        3. HADOOP-15576.003.patch
          25 kB
          Ewan Higgs
        4. HADOOP-15576.004.patch
          35 kB
          Ewan Higgs
        5. HADOOP-15576-005.patch
          49 kB
          Steve Loughran
        6. HADOOP-15576-007.patch
          57 kB
          Steve Loughran
        7. HADOOP-15576-008.patch
          58 kB
          Steve Loughran

        Issue Links

          Activity

            People

              ehiggs Ewan Higgs
              stevel@apache.org Steve Loughran
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: