[HADOOP-15576] S3A Multipart Uploader to work with S3Guard and encryption

• Type: Sub-task
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.2.0
• Component/s: fs/s3
• Target Version/s: 3.2.0
• Assignee: Ewan Higgs
• Reporter: Steve Loughran
• Created: 02/Jul/18 14:00
• Updated: 27/May/19 22:07
• Resolved: 12/Aug/18 14:41

Description

The new Multipart Uploader API of HDFS-13186 needs to work with S3Guard, with the tests to demonstrate this

1. move from low-level calls of S3A client to calls of WriteOperationHelper; adding any new methods are needed there.
2. Tests. the tests of HDFS-13713.
3. test execution, with -DS3Guard, -DAuth

There isn't an S3A version of AbstractSystemMultipartUploaderTest, and even if there was, it might not show that S3Guard was bypassed, because there's no checks that listFiles/listStatus shows the newly committed files.

Similarly, because MPU requests are initiated in S3AMultipartUploader, encryption settings are't picked up. Files being uploaded this way are not being encrypted

Issue Links

• contains: HDFS-13766 HDFS Classes used for implementation of Multipart uploads to move to hadoop-common JAR
• contains: HADOOP-16330 Regression: TestStagingPartitionedJobCommit failing with empty etag list
• depends upon: HDFS-13713 Add specification of Multipart Upload API to FS specification, with contract tests
• depends upon: HDFS-13186 [PROVIDED Phase 2] Multipart Uploader API
• is related to: HADOOP-15631 Remove transient dependency on hadoop-hdfs-client
• is related to: HADOOP-15604 Bulk commits of S3A MPUs place needless excessive load on S3 & S3Guard