Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 2.8.0
    • Fix Version/s: 2.9.0, 3.0.0-alpha4
    • Component/s: fs/s3
    • Labels:
      None
    • Target Version/s:
    • Release Note:
      Hide
      The new encryption options SSE-KMS and especially SSE-C must be considered experimental at present. If you are using SSE-C, problems may arise if the bucket mixes encrypted and unencrypted files. For SSE-KMS, there may be extra throttling of IO, especially with the fadvise=random option. You may wish to request an increase in your KMS IOPs limits.
      Show
      The new encryption options SSE-KMS and especially SSE-C must be considered experimental at present. If you are using SSE-C, problems may arise if the bucket mixes encrypted and unencrypted files. For SSE-KMS, there may be extra throttling of IO, especially with the fadvise=random option. You may wish to request an increase in your KMS IOPs limits.

      Description

      S3 provides 3 types of server-side encryption [1],

      • SSE-S3 (Amazon S3-Managed Keys) [2]
      • SSE-KMS (AWS KMS-Managed Keys) [3]
      • SSE-C (Customer-Provided Keys) [4]

      Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 (HADOOP-10568) – the underlying aws-java-sdk makes that very simple [5]. With native support in aws-java-sdk already available it should be fairly straightforward [6],[7] to support the other two types of SSE with some additional fs.s3a configuration properties.

      [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
      [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
      [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
      [4] http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
      [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
      [6] http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
      [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html

        Attachments

        1. HADOOP-13075-001.patch
          55 kB
          Steve Moist
        2. HADOOP-13075-002.patch
          55 kB
          Steve Moist
        3. HADOOP-13075-003.patch
          59 kB
          Steve Moist
        4. HADOOP-13075-branch2.002.patch
          59 kB
          Steve Moist

          Issue Links

            Activity

              People

              • Assignee:
                moist Steve Moist
                Reporter:
                noslowerdna Andrew Olson
              • Votes:
                4 Vote for this issue
                Watchers:
                22 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: