[HADOOP-13887] Encrypt S3A data client-side with AWS SDK (S3-CSE)

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 3.3.2
• Component/s: fs/s3
• Labels: pull-request-available
• Target Version/s: 3.4.0

Release Note

Adds support for client side encryption in AWS S3,
with keys managed by AWS-KMS.

Read the documentation in encryption.md very, very carefully before
use and consider it unstable.

S3-CSE is enabled in the existing configuration option
"fs.s3a.server-side-encryption-algorithm":

fs.s3a.server-side-encryption-algorithm=CSE-KMS
fs.s3a.server-side-encryption.key=<KMS_KEY_ID>

You cannot enable CSE and SSE in the same client, although
you can still enable a default SSE option in the S3 console.

* Not compatible with S3Guard.
* Filesystem list/get status operations subtract 16 bytes from the length
  of all files >= 16 bytes long to compensate for the padding which CSE
  adds.
* The SDK always warns about the specific algorithm chosen being
  deprecated. It is critical to use this algorithm for ranged
  GET requests to work (i.e. random IO). Ignore.
* Unencrypted files CANNOT BE READ.
  The entire bucket SHOULD be encrypted with S3-CSE.
* Uploading files may be a bit slower as blocks are now
  written sequentially.
* The Multipart Upload API is disabled when S3-CSE is active.

• Assignee: Mehakmeet Singh
• Reporter: Jeeyoung Kim
• Created: 09/Dec/16 23:48
• Updated: 16/Dec/21 09:37
• Resolved: 27/Jul/21 10:32

Description

Expose the client-side encryption option documented in Amazon S3 documentation - http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingClientSideEncryption.html

When backporting, include HADOOP-17817

Issue Links

• depends upon: MAPREDUCE-7182 MapReduce input format/record readers to support S3 select queries
• incorporates: HADOOP-11621 s3a doesn't consider blobs with trailing / and content-length >0 as directories
• is duplicated by: HADOOP-14171 Support for S3 Client Encryption
• is related to: HADOOP-17823 S3A Tests to skip if S3Guard and S3-CSE are enabled.
• is related to: HADOOP-17817 HADOOP-17817. S3A to raise IOE if both S3-CSE and S3Guard enabled
• is related to: HADOOP-17953 S3A: ITestS3AFileContextStatistics test to lookup global or per-bucket configuration for encryption algorithm
• is related to: HADOOP-15006 Encrypt S3A data client-side with Hadoop libraries & Hadoop KMS
• relates to: HADOOP-17871 S3A CSE: minor tuning
• relates to: HADOOP-11202 SequenceFile crashes with client-side encrypted files that are shorter than FileSystem.getStatus(path)
• links to: GitHub Pull Request #2706
• links to: GitHub Pull Request #3292