[HADOOP-13968] S3a FS to support "__magic" path for the special "unmaterialized" writes

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0-beta1
• Component/s: fs/s3
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 10/Jan/17 18:47
• Updated: 21/Mar/18 23:58
• Resolved: 06/Dec/17 13:03

Description

S3AFileSystem to add support for a special path, such as .temp_pending_put/ or similar, which, when used as the base of a path, indicates that the file is actually to be saved to the parent dir, but only via a delayed put commit operation.

At the same time, we may need blocks on some normal fileIO ops under these dirs, especially rename and delete, as this would cause serious problems including data loss and large bills for pending data.

Issue Links

• is part of: HADOOP-13786 Add S3A committers for zero-rename commits to S3 endpoints