[HADOOP-13164] Optimize S3AFileSystem::deleteUnnecessaryFakeDirectories

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.8.0
• Fix Version/s: 2.9.0, 3.0.0-alpha2
• Component/s: fs/s3
• Target Version/s: 2.8.0
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 17/May/16 06:14
• Updated: 05/Mar/20 16:22
• Resolved: 29/Sep/16 16:01

Description

https://github.com/apache/hadoop/blob/27c4e90efce04e1b1302f668b5eb22412e00d033/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AFileSystem.java#L1224

deleteUnnecessaryFakeDirectories is invoked in S3AFileSystem during rename and on outputstream close() to purge any fake directories. Depending on the nesting in the folder structure, it might take a lot longer time as it invokes getFileStatus multiple times. Instead, it should be able to break out of the loop once a non-empty directory is encountered.

Issue Links

• causes: HADOOP-16090 S3A Client to add explicit support for versioned stores
• is depended upon by: HADOOP-11694 Über-jira: S3a phase II: robustness, scale and performance
• is related to: HADOOP-13230 S3A to optionally retain directory markers
• is related to: HADOOP-14594 ITestS3AFileOperationCost::testFakeDirectoryDeletion to uncomment metric assertions