[HADOOP-13914] s3guard: improve S3AFileStatus#isEmptyDirectory handling

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: HADOOP-13345
• Fix Version/s: HADOOP-13345
• Component/s: fs/s3
• Hadoop Flags: Reviewed
• Assignee: Aaron Fabbri Assign to me
• Reporter: Aaron Fabbri
• Created: 17/Dec/16 01:50
• Updated: 10/Mar/17 09:09
• Resolved: 10/Mar/17 09:09

Description

As discussed in HADOOP-13449, proper support for the isEmptyDirectory() flag stored in S3AFileStatus is missing from DynamoDBMetadataStore.

The approach taken by LocalMetadataStore is not suitable for the DynamoDB implementation, and also sacrifices good code separation to minimize S3AFileSystem changes pre-merge to trunk.

I will attach a design doc that attempts to clearly explain the problem and preferred solution. I suggest we do this work after merging the HADOOP-13345 branch to trunk, but am open to suggestions.

I can also attach a patch of a integration test that exercises the missing case and demonstrates a failure with DynamoDBMetadataStore.

Issue Links

• is depended upon by: HADOOP-13998 Merge initial S3guard release into trunk