Uploaded image for project: 'Hadoop Common'
  1. Hadoop Common
  2. HADOOP-15220 Über-jira: S3a phase V: Hadoop 3.2 features
  3. HADOOP-15576

S3A Multipart Uploader to work with S3Guard and encryption

VotersWatch issueWatchersLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments


    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 3.2.0
    • Fix Version/s: 3.2.0
    • Component/s: fs/s3
    • Labels:
    • Target Version/s:


      The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with the tests to demonstrate this

      1. move from low-level calls of S3A client to calls of WriteOperationHelper; adding any new methods are needed there.
      2. Tests. the tests of HDFS-13713.
      3. test execution, with -DS3Guard, -DAuth

      There isn't an S3A version of AbstractSystemMultipartUploaderTest, and even if there was, it might not show that S3Guard was bypassed, because there's no checks that listFiles/listStatus shows the newly committed files.

      Similarly, because MPU requests are initiated in S3AMultipartUploader, encryption settings are't picked up. Files being uploaded this way are not being encrypted


        1. HADOOP-15576-008.patch
          58 kB
          Steve Loughran
        2. HADOOP-15576-007.patch
          57 kB
          Steve Loughran
        3. HADOOP-15576-005.patch
          49 kB
          Steve Loughran
        4. HADOOP-15576.004.patch
          35 kB
          Ewan Higgs
        5. HADOOP-15576.003.patch
          25 kB
          Ewan Higgs
        6. HADOOP-15576.002.patch
          24 kB
          Ewan Higgs
        7. HADOOP-15576.001.patch
          8 kB
          Ewan Higgs

        Issue Links



            • Assignee:
              ehiggs Ewan Higgs
              stevel@apache.org Steve Loughran


              • Created:

                Issue deployment