[HADOOP-13345] S3Guard: Improved Consistency for S3A

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.1
• Fix Version/s: 2.9.0, 3.0.0-beta1
• Component/s: fs/s3
• Target Version/s: 2.9.0

Release Note

S3Guard (pronounced see-guard) is a new feature for the S3A connector to Amazon S3, which uses DynamoDB for a high performance and consistent metadata repository. Essentially: S3Guard caches directory information, so your S3A clients get faster lookups and resilience to inconsistency between S3 list operations and the status of objects. When files are created, with S3Guard, they'll always be found.

S3Guard does not address update consistency: if a file is updated, while the directory information will be updated, calling open() on the path may still return the old data. Similarly, deleted objects may also potentially be opened.

Please consult the S3Guard documentation in the Amazon S3 section of our documentation.

Note: part of this update includes moving to a new version of the AWS SDK 1.11, one which includes the Dynamo DB client and its a shaded version of Jackson 2. The large aws-sdk-bundle JAR is needed to use the S3A client with or without S3Guard enabled. The good news: because Jackson is shaded, there will be no conflict between any Jackson version used in your application and that which the AWS SDK needs.

• Assignee: Chris Nauroth
• Reporter: Chris Nauroth
• Created: 06/Jul/16 21:23
• Updated: 03/Feb/18 03:08
• Resolved: 01/Sep/17 14:49

Description

This issue proposes S3Guard, a new feature of S3A, to provide an option for a stronger consistency model than what is currently offered. The solution coordinates with a strongly consistent external store to resolve inconsistencies caused by the S3 eventual consistency model.

Issue Links

• depends upon: HADOOP-14432 S3A copyFromLocalFile to be robust, tested
• depends upon: HADOOP-13912 S3a Multipart Committer (avoid rename)
• depends upon: HADOOP-13852 hadoop build to allow hadoop version property to be explicitly set
• incorporates: HADOOP-14838 backport S3guard to branch-2
• is depended upon by: SPARK-17593 list files on s3 very slow
• is depended upon by: HADOOP-13204 Über-jira: S3a phase III: scale and tuning
• is related to: HADOOP-11487 FileNotFound on distcp to s3n/s3a due to creation inconsistency
• is related to: SPARK-18883 FileNotFoundException on _temporary directory
• is superceded by: HADOOP-14825 Über-JIRA: S3Guard Phase II: Hadoop 3.1 features
• supercedes: HADOOP-14161 Failed to rename file in S3A during FileOutputFormat commitTask