Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15220 Über-jira: S3a phase V: Hadoop 3.2 features
  3. HADOOP-15576

S3A Multipart Uploader to work with S3Guard and encryption

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.0
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:

      Description

      The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with the tests to demonstrate this

      1. move from low-level calls of S3A client to calls of WriteOperationHelper; adding any new methods are needed there.
      2. Tests. the tests of HDFS-13713.
      3. test execution, with -DS3Guard, -DAuth

      There isn't an S3A version of AbstractSystemMultipartUploaderTest, and even if there was, it might not show that S3Guard was bypassed, because there's no checks that listFiles/listStatus shows the newly committed files.

      Similarly, because MPU requests are initiated in S3AMultipartUploader, encryption settings are't picked up. Files being uploaded this way are not being encrypted

        Attachments

        1. HADOOP-15576-008.patch
          58 kB
          Steve Loughran
        2. HADOOP-15576-007.patch
          57 kB
          Steve Loughran
        3. HADOOP-15576-005.patch
          49 kB
          Steve Loughran
        4. HADOOP-15576.004.patch
          35 kB
          Ewan Higgs
        5. HADOOP-15576.003.patch
          25 kB
          Ewan Higgs
        6. HADOOP-15576.002.patch
          24 kB
          Ewan Higgs
        7. HADOOP-15576.001.patch
          8 kB
          Ewan Higgs

          Issue Links

            Activity

              People

              • Assignee:
                ehiggs Ewan Higgs
                Reporter:
                stevel@apache.org Steve Loughran
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: