[HADOOP-13972] ADLS to support per-store configuration

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0-alpha2
• Fix Version/s: 3.1.0, 2.10.0, 2.9.1, 2.8.4, 3.0.3
• Component/s: fs/adl
• Target Version/s: 3.1.0
• Assignee: Sharad Sonker
• Reporter: John Zhuge
• Created: 11/Jan/17 07:53
• Updated: 30/Nov/20 11:26
• Resolved: 15/Feb/18 16:28

Description

Useful when distcp needs to access 2 Data Lake stores with different SPIs.

Of course, a workaround is to grant the same SPI access permission to both stores, but sometimes it might not be feasible.

One idea is to embed the store name in the configuration property names, e.g., dfs.adls.oauth2.<store>.client.id. Per-store keys will be consulted first, then fall back to the global keys.

Issue Links

• depends upon: HADOOP-13336 S3A to support per-bucket configuration
• depends upon: HADOOP-17401 GCS to support per-bucket configuration
• is duplicated by: HADOOP-15238 ADLS to support per-store configuration: Yetus patch checker
• relates to: HADOOP-14507 extend per-bucket secret key config with explicit getPassword() on fs.s3a.$bucket.secret.key
• links to: GitHub Pull Request #339

Sub-Tasks

• Back port HADOOP-13972 to 2.8.1 and 2.8.3: Resolved