[HADOOP-16090] S3A Client to add explicit support for versioned stores

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Abandoned
• Affects Version/s: 2.8.1
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Dmitri Chmelev
• Created: 01/Feb/19 02:40
• Updated: 02/Jun/20 13:22
• Resolved: 02/Jun/20 13:22

Description

The fix to avoid calls to getFileStatus() for each path component in deleteUnnecessaryFakeDirectories() (HADOOP-13164) results in accumulation of delete markers in versioned S3 buckets. The above patch replaced getFileStatus() checks with a single batch delete request formed by generating all ancestor keys formed from a given path. Since the delete request is not checking for existence of fake directories, it will create a delete marker for every path component that did not exist (or was previously deleted). Note that issuing a DELETE request without specifying a version ID will always create a new delete marker, even if one already exists (AWS S3 Developer Guide)

Since deleteUnnecessaryFakeDirectories() is called as a callback on successful writes and on renames, delete markers accumulate rather quickly and their rate of accumulation is inversely proportional to the depth of the path. In other words, directories closer to the root will have more delete markers than the leaves.

This behavior negatively impacts performance of getFileStatus() operation when it has to issue listObjects() request (especially v1) as the delete markers have to be examined when the request searches for first current non-deleted version of an object following a given prefix.

I did a quick comparison against 3.x and the issue is still present: https://github.com/apache/hadoop/blob/trunk/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L2947

 

Issue Links

• is caused by: HADOOP-13164 Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories
• is related to: HADOOP-16134 Add S3AWriteOpContext for write ops; pass in statistics and other settings
• is related to: HADOOP-16942 S3A creating folder level delete markers
• is superceded by: HADOOP-13230 S3A to optionally retain directory markers
• relates to: HADOOP-15625 S3A input stream to use etags/version number to detect changed source files
• relates to: HADOOP-16085 S3Guard: use object version or etags to protect against inconsistent read after replace/overwrite
• relates to: HADOOP-16202 Enhance openFile() for better read performance against object stores
• links to: GitHub Pull Request #621