[HADOOP-18281] Tune S3A storage class support

• Type: Improvement
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.3.5
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 08/Jun/22 19:03
• Updated: 12/Dec/22 11:35

Description

Followup to HADOOP-12020, with work/review from rebasing HADOOP-17833 atop it.

• Can we merge ITestS3AHugeFilesStorageClass into one of the existing test cases? just because it is slow...ideally we want as few of those as possible, even if by testing multiple things at the same we break the rules of testing.
• move setting the storage class into
  setOptionalMultipartUploadRequestParameters and setOptionalPutRequestParameters
• both newPutObjectRequest() calls to set storage class
• docs to list the valid option strings. I had to delve into the AWS SDK to work them out

Once HADOOP-17833 is in, make this a new option something which can be explicitly used in createFile().
I've updated PutObjectOptions to pass a value around, and made sure it gets down to to the request factory. that leaves

• setting the storage class from the options CreateFileBuilder
• testing!
• doc update

Issue Links

• is blocked by: HADOOP-17833 Improve Magic Committer Performance
• is related to: HADOOP-12020 Support configuration of different S3 storage classes