[HADOOP-18339] S3A storage class option only picked up when buffering writes to disk

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.5
• Fix Version/s: 3.4.0, 3.3.5
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0, 3.3.5
• Hadoop Flags: Reviewed
• Assignee: Monthon Klongklaew
• Reporter: Steve Loughran
• Created: 15/Jul/22 10:20
• Updated: 28/Jan/24 00:25
• Resolved: 01/Sep/22 17:19

Description

when you switch s3a output stream buffering to heap or byte buffer, the storage class option isn't added to the put request

  <property>
    <name>fs.s3a.fast.upload.buffer</name>
    <value>bytebuffer</value>
  </property>

and the ITestS3AStorageClass tests fail.

java.lang.AssertionError: [Storage class of object s3a://stevel-london/test/testCreateAndCopyObjectWithStorageClassGlacier/file1] 
Expecting:
 <null>
to be equal to:
 <"glacier">
ignoring case considerations

	at org.apache.hadoop.fs.s3a.ITestS3AStorageClass.assertObjectHasStorageClass(ITestS3AStorageClass.java:215)
	at org.apache.hadoop.fs.s3a.ITestS3AStorageClass.testCreateAndCopyObjectWithStorageClassGlacier(ITestS3AStorageClass.java:129)

we noticed this in a code review; the request factory only sets the option when the source is a file, not memory.

proposed: parameterize the test suite on disk/byte buffer, then fix

Issue Links

• is caused by: HADOOP-12020 Support configuration of different S3 storage classes
• is depended upon by: HADOOP-16259 Distcp to set S3 Storage Class
• is related to: HADOOP-12020 Support configuration of different S3 storage classes
• links to: GitHub Pull Request #4669