[HADOOP-13345] S3Guard: Improved Consistency for S3A

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 2.9.0, 3.0.0-beta1
• Component/s: fs/s3
• Target Version/s: 2.9.0

Release Note

S3Guard (pronounced see-guard) is a new feature for the S3A connector to Amazon S3, which uses DynamoDB for a high performance and consistent metadata repository. Essentially: S3Guard caches directory information, so your S3A clients get faster lookups and resilience to inconsistency between S3 list operations and the status of objects. When files are created, with S3Guard, they'll always be found.

S3Guard does not address update consistency: if a file is updated, while the directory information will be updated, calling open() on the path may still return the old data. Similarly, deleted objects may also potentially be opened.

Please consult the S3Guard documentation in the Amazon S3 section of our documentation.

Note: part of this update includes moving to a new version of the AWS SDK 1.11, one which includes the Dynamo DB client and its a shaded version of Jackson 2. The large aws-sdk-bundle JAR is needed to use the S3A client with or without S3Guard enabled. The good news: because Jackson is shaded, there will be no conflict between any Jackson version used in your application and that which the AWS SDK needs.

• Assignee: Chris Nauroth
• Reporter: Chris Nauroth
• Created: 06/Jul/16 21:23
• Updated: 03/Feb/18 03:08
• Resolved: 01/Sep/17 14:49

Description

This issue proposes S3Guard, a new feature of S3A, to provide an option for a stronger consistency model than what is currently offered. The solution coordinates with a strongly consistent external store to resolve inconsistencies caused by the S3 eventual consistency model.

Issue Links

• depends upon: HADOOP-14432 S3A copyFromLocalFile to be robust, tested
• depends upon: HADOOP-13912 S3a Multipart Committer (avoid rename)
• depends upon: HADOOP-13852 hadoop build to allow hadoop version property to be explicitly set
• incorporates: HADOOP-14838 backport S3guard to branch-2
• is depended upon by: SPARK-17593 list files on s3 very slow
• is depended upon by: HADOOP-13204 Über-jira: S3a phase III: scale and tuning
• is related to: HADOOP-11487 FileNotFound on distcp to s3n/s3a due to creation inconsistency
• is related to: SPARK-18883 FileNotFoundException on _temporary directory
• is superceded by: HADOOP-14825 Über-JIRA: S3Guard Phase II: Hadoop 3.1 features
• supercedes: HADOOP-14161 Failed to rename file in S3A during FileOutputFormat commitTask

Sub-Tasks

• Support running isolated unit tests separate from AWS integration tests.: Resolved
• S3Guard: Define MetadataStore interface.: Resolved
• S3Guard: Implement DynamoDBMetadataStore.: Resolved
• S3Guard: Implement access policy providing strong consistency with S3 as source of truth.: Closed
• S3Guard: Implement access policy using metadata store as source of truth.: Closed
• S3Guard: Implement access policy for intra-client consistency with in-memory metadata store.: Resolved
• S3Guard: Instrument new functionality with Hadoop metrics.: Resolved
• S3Guard: Write end user docs, change table autocreate default.: Resolved
• S3Guard: create basic contract tests for MetadataStore implementations: Resolved
• S3Guard: implement move() for LocalMetadataStore, add unit tests: Resolved
• S3Guard: Allow execution of all S3A integration tests with S3Guard enabled.: Resolved
• S3Guard: S3AFileSystem Integration with MetadataStore: Resolved
• S3Guard: Provide command line tools to manipulate metadata store.: Resolved
• Change PathMetadata to hold S3AFileStatus instead of FileStatus.: Resolved
• S3Guard: better support for multi-bucket access: Resolved
• S3Guard: add delete tracking: Resolved
• s3guard: add inconsistency injection, integration tests: Resolved
• s3guard to log choice of metadata store at debug: Resolved
• S3Guard: fix TestDynamoDBMetadataStore when fs.s3a.s3guard.ddb.table is set: Resolved
• s3guard: ITestS3AFileOperationCost.testFakeDirectoryDeletion failure: Resolved
• dynamodb dependency -> compile: Resolved
• DynamoDBMetadataStore to handle DDB throttling failures through retry policy: Resolved
• tune dynamodb client & tests: Resolved
• s3guard: improve S3AFileStatus#isEmptyDirectory handling: Resolved
• S3Guard: Existing tables may not be initialized correctly in DynamoDBMetadataStore: Resolved
• S3Guard: NPE when table is already populated in dynamodb and user specifies "fs.s3a.s3guard.ddb.table.create=false": Closed
• S3AGuard: Use BatchWriteItem in DynamoDBMetadataStore#put(): Resolved
• S3Guard: S3AFileSystem::listLocatedStatus() to employ MetadataStore: Resolved
• S3Guard: DynamoDBMetadataStore#move() could be throwing exception due to BatchWriteItem limits: Resolved
• Mock bucket locations in MockS3ClientFactory: Resolved
• S3guard: replace dynamo.describe() call in init with more efficient query: Closed
• Initialize DynamoDBMetadataStore without associated S3AFileSystem: Resolved
• Add ability to start DDB local server in every test: Resolved
• s3guard: add a version marker to every table: Resolved
• S3Guard CLI: Add documentation: Resolved
• s3guard cli: make tests easier to run and address failure: Resolved
• Merge initial S3guard release into trunk: Resolved
• cli to list info about a bucket (S3guard or not): Resolved
• Handled dynamo exceptions in translateException: Resolved
• S3Guard: fix multi-bucket integration tests: Resolved
• Optimize dirListingUnion: Resolved
• S3Guard: DynamoDBMetadataStore logs nonsense region: Resolved
• Implicitly creating DynamoDB table ignores endpoint config: Resolved
• S3Guard: intermittent duplicate item keys failure: Resolved
• CLI command to prune old metadata: Resolved
• Metastore destruction test creates table without version marker: Resolved
• S3Guard: link docs from index, fix typos: Resolved
• Fix breaking link in s3guard.md: Resolved
• Drop unnecessary type assertion and cast: Resolved
• Allow users to specify region for DynamoDB table instead of endpoint: Resolved
• Rethink S3GuardTool options: Resolved
• s3guard: regression in dirListingUnion: Resolved
• ITestS3GuardListConsistency fails intermittently: Resolved
• In S3AFileSystem, make getAmazonClient() package private; export getBucketLocation(): Resolved
• s3guard tool tests aren't isolated; can't run in parallel: Resolved
• ITestS3ACredentialsInURL sometimes fails: Resolved
• Simplify DynamoDBClientFactory for creating Amazon DynamoDB clients: Resolved
• s3guard: CLI diff non-empty after import on new table: Resolved
• Ensure GenericOptionParser is used for S3Guard CLI: Resolved
• Add S3Guard.dirListingUnion in S3AFileSystem#listFiles, listLocatedStatus: Closed
• S3GuardTool tests should not run if S3Guard is not set up: Resolved
• S3Guard: import does not import empty directory: Resolved
• Add validation of DynamoDB region: Resolved
• DynamoDB client should waitForActive on existing tables: Resolved
• Add s3guardtool dump command: Resolved
• S3Guard: DynamoDBMetadataStore::move() should populate ancestor directories of destination paths: Resolved
• S3Guard: S3AFileSystem::rename() should move non-listed sub-directory entries in metadata store: Resolved
• S3Guard: ITestS3AConcurrentOps is not cleaning up test data: Resolved
• TestS3GuardTool hangs/fails when offline: it's an IT test: Resolved
• S3Guard: S3AFileSystem::listFiles() to employ MetadataStore: Resolved
• S3Guard: DynamoDBMetadata::prune() should self interrupt correctly: Resolved
• TestDynamoDBMetadataStore is broken unless we can fail faster without a table version: Resolved
• ITestS3GuardListConsistency failure w/ Local, authoritative metadata store: Resolved
• S3Guard: S3GuardTool to support provisioning existing metadata store: Resolved
• s3guard will set file length to -1 on a putObjectDirect(stream, -1) call: Resolved
• ITestS3GuardConcurrentOps.testConcurrentTableCreations fails without table name configured: Resolved
• Play nice with ITestS3AEncryptionSSEC: Resolved
• create() does not notify metadataStore of parent directories or ensure they're not existing files: Resolved
• S3Guard: Improve FNFE message when opening a stream: Resolved
• make InconsistentAmazonS3Client usable in downstream tests: Resolved
• Ensure deleted parent directory tombstones are overwritten when implicitly recreated: Resolved
• DirListingMetadata precondition failure messages to include path at fault: Resolved
• s3guard w/ failure injection: listStatus fails after renaming file into directory: Resolved
• ITestS3GuardConcurrentOps requires explicit DynamoDB table name to be configured: Resolved
• Findbugs warning in LocalMetadataStore: Resolved
• ProvidedFileStatusIterator#next() may throw IndexOutOfBoundsException: Resolved
• simplify mkdirs() after S3Guard delete tracking change: Resolved
• InconsistentAmazonS3Client adds extra paths to listStatus() after delete.: Resolved
• ITestS3GuardListConsistency is too slow: Resolved
• S3Guard: issues running parallel tests w/ S3N: Resolved
• LocalDynamoDB missing from latest AWS SDK releases: Resolved
• S3Guard: optimize create codepath: Resolved
• add a predicate/option to probe an S3A FS for being consistent: Resolved
• ITestS3GuardConcurrentOps failing with -Ddynamodblocal -Ds3guard: Resolved
• ITestS3AEncryptionSSEC failing in parallel s3guard runs: Resolved
• Review S3guard docs & code prior to merge: Resolved
• S3Guard premerge changes: java 7 build & test tuning: Resolved
• s3guard diff demand creates a new table: Resolved
• hadoop-aws shell profile not being built: Resolved
• S3Guard: handle provisioning failure through backoff & retry (& metrics): Resolved
• s3guard usage calls function incorrectly: Resolved
• backport S3guard to branch-2: Resolved