Hadoop Common
  1. Hadoop Common
  2. HADOOP-9680

Extend S3FS and S3NativeFS to work with AWS IAM Temporary Security Credentials

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Minor Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.0, 2.1.0-beta
    • Fix Version/s: None
    • Component/s: fs/s3
    • Labels:
      None

      Description

      Here is a patch in unified diff format to enable Amazon Web Services IAM Temporary Security Credentials secured interactions with S3 from Hadoop.

      It bumps the JetS3t release version up to 0.9.0.

      To use a temporary security credential set, you need to provide the following properties, depending on the implementation (s3 or s3native):

      fs.s3.awsAccessKeyId or fs.s3n.awsAccessKeyId - the temporary access key id issued by AWS IAM
      fs.s3.awsSecretAccessKey or fs.s3n.awsSecretAccessKey - the temporary secret access key issued by AWS IAM
      fs.s3.awsSessionToken or fs.s3n.awsSessionToken - the session ticket issued by AWS IAM along with the temporary key
      fs.s3.awsTokenFriendlyName or fs.s3n.awsTokenFriendlyName - any string

        Issue Links

          Activity

          Hide
          Timothy St. Clair added a comment -
          Show
          Timothy St. Clair added a comment - Seems directly related to--> https://issues.apache.org/jira/browse/HADOOP-9623
          Hide
          Robert Gibbon added a comment -

          I took a look at your patch in HADOOP-9623. Some comments:

          • Bucket keyspace listings running over a s3-native fs will be broken by your patch, they make use of the method org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(String key) when querying S3 for the given URI; if the URI does not correspond to a single key in a bucket (ie a single object) an exception is thrown. In the above mentioned method, the exception's Message property was being parsed for the string "ResponseCode=404" to interpret that the URI is not a single key. If the condition is met, it returns null. Its a horrible piece of code and a very poorly defined contract with the calling party. It is also broken by jets3t 0.9.0, which doesn't pass back that message anymore in that situation. I adapted it to look at the ResponseCode property for the integer 404 instead, but someone who knows that code better than me would do a good deed to fix it more sustainably.
          • I needed to upgrade jets3t to 0.9.0 because I need support for AWS IAM federated access tokens (temporary, time limited access credentials, tied to a session ticket). I don't see any support for that in the patch in HADOOP-9623, for me its of no value unless it supports temp security tokens.

          I think we're aligned on the need for an uprev of the jets3t implementation in hadoop.
          HTH

          Show
          Robert Gibbon added a comment - I took a look at your patch in HADOOP-9623 . Some comments: Bucket keyspace listings running over a s3-native fs will be broken by your patch, they make use of the method org.apache.hadoop.fs.s3native.Jets3tNativeFileSystemStore.retrieveMetadata(String key) when querying S3 for the given URI; if the URI does not correspond to a single key in a bucket (ie a single object) an exception is thrown. In the above mentioned method, the exception's Message property was being parsed for the string "ResponseCode=404" to interpret that the URI is not a single key. If the condition is met, it returns null. Its a horrible piece of code and a very poorly defined contract with the calling party. It is also broken by jets3t 0.9.0, which doesn't pass back that message anymore in that situation. I adapted it to look at the ResponseCode property for the integer 404 instead, but someone who knows that code better than me would do a good deed to fix it more sustainably. I needed to upgrade jets3t to 0.9.0 because I need support for AWS IAM federated access tokens (temporary, time limited access credentials, tied to a session ticket). I don't see any support for that in the patch in HADOOP-9623 , for me its of no value unless it supports temp security tokens. I think we're aligned on the need for an uprev of the jets3t implementation in hadoop. HTH
          Hide
          Daisuke Miyamoto added a comment -

          +1

          Show
          Daisuke Miyamoto added a comment - +1

            People

            • Assignee:
              Unassigned
              Reporter:
              Robert Gibbon
            • Votes:
              1 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development