[HADOOP-13075] Add support for SSE-KMS and SSE-C in s3a filesystem

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-alpha4
• Component/s: fs/s3
• Target Version/s: 2.9.0

Release Note

The new encryption options SSE-KMS and especially SSE-C must be considered experimental at present. If you are using SSE-C, problems may arise if the bucket mixes encrypted and unencrypted files. For SSE-KMS, there may be extra throttling of IO, especially with the fadvise=random option. You may wish to request an increase in your KMS IOPs limits.

• Assignee: Steve Moist
• Reporter: Andrew Olson
• Created: 29/Apr/16 19:54
• Updated: 01/Mar/19 18:04
• Resolved: 11/Feb/17 23:08

Description

S3 provides 3 types of server-side encryption [1],

• SSE-S3 (Amazon S3-Managed Keys) [2]
• SSE-KMS (AWS KMS-Managed Keys) [3]
• SSE-C (Customer-Provided Keys) [4]

Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 (HADOOP-10568) – the underlying aws-java-sdk makes that very simple [5]. With native support in aws-java-sdk already available it should be fairly straightforward [6],[7] to support the other two types of SSE with some additional fs.s3a configuration properties.

[1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
[2] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
[3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
[4] http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
[5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
[6] http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
[7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html

Issue Links

• breaks: HADOOP-14102 Relax error message assertion in S3A test ITestS3AEncryptionSSEC
• breaks: HADOOP-14120 needless S3AFileSystem.setOptionalPutRequestParameters in S3ABlockOutputStream putObject()
• is depended upon by: HADOOP-14324 Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics
• is depended upon by: DRILL-5536 Support AWS S3 SSE-KMS encrypted objects querying
• relates to: HADOOP-14305 S3A SSE tests won't run in parallel: Bad request in directory GetFileStatus
• links to: GitHub Pull Request #110
• links to: GitHub Pull Request #110
• links to: GitHub Pull Request #113
• links to: GitHub Pull Request #113
• links to: GitHub Pull Request #183