[HADOOP-13407] s3a directory housekeeping operations to be done in async thread

• Type: Sub-task
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 2.8.0, 3.1.0, 2.9.1, 3.0.3
• Component/s: fs/s3
• Assignee: Unassigned
• Reporter: Steve Loughran
• Created: 22/Jul/16 13:00
• Updated: 04/Mar/21 17:42
• Resolved: 04/Mar/21 17:42

Description

Some of the delays on s3a calls are due to cleaning up parent pseudo directories; repeated getParent/GET calls to look for the entries, then to delete them.

We could possibly make this asynchronous; the core semantics would be retained, just the cleanup delayed.

Risks?

1. while the cleanup is in progress, getFileStatus of parent dirs could imply that the parent dir is still empty
2. failure
   of course, these risks exist today. We really need an s3a fsck

Issue Links

• relates to: HADOOP-13330 Parallelize S3A directory deletes