[HADOOP-14507] extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key

• Type: Sub-task
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 3.1.0
• Component/s: fs/s3
• Target Version/s: 3.1.0
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 08/Jun/17 10:28
• Updated: 26/Jul/18 01:27
• Resolved: 16/Feb/18 16:37

Description

Per-bucket jceks support turns out to be complex as you have to manage multiple jecks files & configure the client to ask for the right one. This is because we're calling Configuration.getPassword{"fs,s3a.secret.key".

If before that, we do a check for the explict id, key, session key in the properties fs.s3a.$bucket.secret ( & c), we could have a single JCEKs file with all the secrets for different bucket. You would only need to explicitly point the base config to the secrets file, and the right credentials would be picked up, if set

Issue Links

• contains: HADOOP-14723 reinstate URI parameter in AWSCredentialProvider constructors
• depends upon: HADOOP-14723 reinstate URI parameter in AWSCredentialProvider constructors
• is broken by: HADOOP-14135 Remove URI parameter in AWSCredentialProvider constructors
• is depended upon by: HADOOP-14614 Hive doesn't let s3a patch the credential provider path
• is related to: HADOOP-14821 Executing the command 'hdfs -Dhadoop.security.credential.provider.path=file1.jceks,file2.jceks' fails if permission is denied to some files
• is related to: HADOOP-13972 ADLS to support per-store configuration
• relates to: HADOOP-14324 Refine S3 server-side-encryption key as encryption secret; improve error reporting and diagnostics
• supercedes: HADOOP-14625 error message in S3AUtils.getServerSideEncryptionKey() needs to expand property constant