[HADOOP-13876] S3Guard: better support for multi-bucket access

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HADOOP-13345
• Component/s: fs/s3
• Target Version/s: HADOOP-13345
• Assignee: Aaron Fabbri
• Reporter: Aaron Fabbri
• Created: 09/Dec/16 00:12
• Updated: 30/Nov/20 11:26
• Resolved: 04/Feb/17 01:27

Description

HADOOP-13449 adds support for DynamoDBMetadataStore.

The code currently supports two options for choosing DynamoDB table names:
1. Use name of each s3 bucket and auto-create a DynamoDB table for each.
2. Configure a table name in the fs.s3a.s3guard.ddb.table parameter.

However, if a user sets fs.s3a.s3guard.ddb.table and accesses multiple buckets, DynamoDBMetadataStore does not properly differentiate between paths belonging to different buckets. For example, it would treat s3a://bucket-a/path1 as the same as s3a://bucket-b/path1.

Goals for this JIRA:

• Allow for a "one DynamoDB table per cluster" configuration. If a user accesess multiple buckets with that single table, it should work correctly.
• Explain which credentials are used for DynamoDB. Currently each S3AFileSystem has its own DynamoDBMetadataStore, which uses the credentials from the S3A fs. We at least need to document this behavior.
• Document any other limitations etc. in the s3guard.md site doc.

Issue Links

• depends upon: HADOOP-13589 S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.
• is cloned by: HADOOP-14013 S3Guard: fix multi-bucket integration tests
• is depended upon by: HADOOP-13998 Merge initial S3guard release into trunk
• is related to: HADOOP-13336 S3A to support per-bucket configuration
• is related to: HADOOP-17401 GCS to support per-bucket configuration