XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.8.0
    • 2.9.0, 3.0.0-alpha4
    • fs/s3
    • None
    • Hide
      The new encryption options SSE-KMS and especially SSE-C must be considered experimental at present. If you are using SSE-C, problems may arise if the bucket mixes encrypted and unencrypted files. For SSE-KMS, there may be extra throttling of IO, especially with the fadvise=random option. You may wish to request an increase in your KMS IOPs limits.
      Show
      The new encryption options SSE-KMS and especially SSE-C must be considered experimental at present. If you are using SSE-C, problems may arise if the bucket mixes encrypted and unencrypted files. For SSE-KMS, there may be extra throttling of IO, especially with the fadvise=random option. You may wish to request an increase in your KMS IOPs limits.

    Description

      S3 provides 3 types of server-side encryption [1],

      • SSE-S3 (Amazon S3-Managed Keys) [2]
      • SSE-KMS (AWS KMS-Managed Keys) [3]
      • SSE-C (Customer-Provided Keys) [4]

      Of which the S3AFileSystem in hadoop-aws only supports opting into SSE-S3 (HADOOP-10568) – the underlying aws-java-sdk makes that very simple [5]. With native support in aws-java-sdk already available it should be fairly straightforward [6],[7] to support the other two types of SSE with some additional fs.s3a configuration properties.

      [1] http://docs.aws.amazon.com/AmazonS3/latest/dev/serv-side-encryption.html
      [2] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingServerSideEncryption.html
      [3] http://docs.aws.amazon.com/AmazonS3/latest/dev/UsingKMSEncryption.html
      [4] http://docs.aws.amazon.com/AmazonS3/latest/dev/ServerSideEncryptionCustomerKeys.html
      [5] http://docs.aws.amazon.com/AmazonS3/latest/dev/SSEUsingJavaSDK.html
      [6] http://docs.aws.amazon.com/AmazonS3/latest/dev/kms-using-sdks.html#kms-using-sdks-java
      [7] http://docs.aws.amazon.com/AmazonS3/latest/dev/sse-c-using-java-sdk.html

      Attachments

        1. HADOOP-13075-001.patch
          55 kB
          Steve Moist
        2. HADOOP-13075-002.patch
          55 kB
          Steve Moist
        3. HADOOP-13075-003.patch
          59 kB
          Steve Moist
        4. HADOOP-13075-branch2.002.patch
          59 kB
          Steve Moist

        Issue Links

          Activity

            People

              moist Steve Moist
              noslowerdna Andrew Olson
              Votes:
              4 Vote for this issue
              Watchers:
              22 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: