[HADOOP-12020] Support configuration of different S3 storage classes

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.7.0
• Fix Version/s: 3.3.5
• Component/s: fs/s3
• Labels: pull-request-available

Environment

Hadoop on AWS

• Target Version/s: 3.4.0
• Assignee: Monthon Klongklaew
• Reporter: Yann Landrin-Schweitzer
• Created: 22/May/15 19:55
• Updated: 13/Oct/22 12:14
• Resolved: 08/Jun/22 19:02

Description

Amazon S3 uses, by default, the NORMAL_STORAGE class for s3 objects.
This offers, according to Amazon's material, 99.99999999% reliability.
For many applications, however, the 99.99% reliability offered by the REDUCED_REDUNDANCY storage class is amply sufficient, and comes with a significant cost saving.

HDFS, when using the legacy s3n protocol, or the new s3a scheme, should support overriding the default storage class of created s3 objects so that users can take advantage of this cost benefit.

This would require minor changes of the s3n and s3a drivers, using
a configuration property fs.s3n.storage.class to override the default storage when desirable.

This override could be implemented in Jets3tNativeFileSystemStore with:
S3Object object = new S3Object(key);
...
if(storageClass!=null) object.setStorageClass(storageClass);

It would take a more complex form in s3a, e.g. setting:
InitiateMultipartUploadRequest initiateMPURequest =
new InitiateMultipartUploadRequest(bucket, key, om);
if(storageClass !=null )

{ initiateMPURequest = initiateMPURequest.withStorageClass(storageClass); }

and similar statements in various places.

Issue Links

• breaks: HADOOP-18292 s3a storage class reduced redundancy breaks s3 select tests
• causes: HADOOP-18371 s3a FS init logs at warn if fs.s3a.create.storage.class is unset
• causes: HADOOP-18339 S3A storage class option only picked up when buffering writes to disk
• depends upon: HADOOP-13050 Upgrade to AWS SDK 1.11.45
• is duplicated by: HADOOP-14326 S3A to support S3 reduced redundancy storage
• is duplicated by: HADOOP-16259 Distcp to set S3 Storage Class
• is related to: HADOOP-16259 Distcp to set S3 Storage Class
• relates to: HADOOP-14837 Handle S3A "glacier" data
• relates to: HADOOP-18339 S3A storage class option only picked up when buffering writes to disk
• relates to: HADOOP-18281 Tune S3A storage class support
• links to: GitHub Pull Request #3877
• links to: GitHub Pull Request #3877