[HADOOP-14556] S3A to support Delegation Tokens

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.3.0
• Fix Version/s: 3.3.0
• Component/s: fs/s3
• Target Version/s: 3.3.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 20/Jun/17 16:38
• Updated: 09/Jul/22 17:10
• Resolved: 14/Jan/19 18:01

Description

S3A to support delegation tokens where

• an authenticated client can request a token via FileSystem.getDelegationToken()
• Amazon's token service is used to request short-lived session secret & id; these will be saved in the token and marshalled with jobs
• A new authentication provider will look for a token for the current user and authenticate the user if found

This will not support renewals; the lifespan of a token will be limited to the initial duration. Also, as you can't request an STS token from a temporary session, IAM instances won't be able to issue tokens.

Issue Links

• breaks: HADOOP-18330 S3AFileSystem removes Path when calling createS3Client
• causes: HADOOP-16033 hamcrest-library declaration in hadoop-aws to be scoped test
• contains: HADOOP-16033 hamcrest-library declaration in hadoop-aws to be scoped test
• contains: HADOOP-15806 Move ITestS3AMiniYarnCluster to S3A committers
• contains: HADOOP-15091 S3aUtils.getEncryptionAlgorithm() always logs@Debug "Using SSE-C"
• depends upon: HADOOP-15583 Stabilize S3A Assumed Role support
• depends upon: HADOOP-14723 reinstate URI parameter in AWSCredentialProvider constructors
• depends upon: HADOOP-15808 Harden Token service loader use
• incorporates: HDFS-13951 HDFS DelegationTokenFetcher can't print non-HDFS tokens in a tokenfile
• is blocked by: HADOOP-14833 Remove s3a user:secret authentication
• is depended upon by: HIVE-16913 Support per-session S3 credentials
• is related to: HADOOP-15862 ABFS to support a Delegation Token provider which marshalls current Oauth secrets
• is related to: HADOOP-15921 UGI.createLoginUser to log token filename & token identifiers on load
• is related to: HADOOP-16164 S3aDelegationTokens to add accessor for tests to get at the token binding
• is related to: HADOOP-13276 S3a operations keep retrying if the password is wrong
• is related to: HADOOP-15650 Add custom InstanceProfileCredentialsProvider with more resilience to throttling
• is related to: HADOOP-16079 Token.toString faulting if any token listed can't load.
• is related to: HADOOP-16068 ABFS Authentication and Delegation Token plugins to optionally be bound to specific URI of the store
• is related to: HADOOP-12770 KMSClientProvider addDelegationTokens won't add if the credentials contain an expired one
• is related to: HBASE-20774 FSHDFSUtils#isSameHdfs doesn't handle S3 filesystems correctly.
• is related to: HADOOP-16658 S3A connector does not support including the token renewer in the token identifier
• is related to: HDFS-14060 HDFS fetchdt command to return error codes on success/failure
• is related to: MAPREDUCE-7154 TokenCache.obtainTokensForNamenodes() to get DTs even when security is off
• relates to: HADOOP-15672 add s3guard CLI command to generate session keys for an assumed role
• relates to: MAPREDUCE-7149 javadocs for FileInputFormat and OutputFormat to mention DT collection
• supercedes: HADOOP-15672 add s3guard CLI command to generate session keys for an assumed role