[HADOOP-18793] S3A StagingCommitter does not clean up staging-uploads directory

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.2.2
• Fix Version/s: 3.3.9
• Component/s: fs/s3
• Labels: pull-request-available
• Assignee: Harunobu Daikoku
• Reporter: Harunobu Daikoku
• Created: 06/Jul/23 10:39
• Updated: 10/Jul/23 13:36
• Resolved: 08/Jul/23 13:04

Description

When setting up StagingCommitter and its internal FileOutputCommitter, a temporary directory that holds MPU information will be created on the default FS, which by default is to be /user/${USER}/tmp/staging/${USER}/${UUID}/staging-uploads.

On a successful job commit, its child directory (_temporary) will be cleaned up properly, but ${UUID}/staging-uploads will remain.

This will result in having too many empty ${UUID}/staging-uploads directories under /user/${USER}/tmp/staging/${USER}, and will eventually cause an issue in an environment where the max number of items in a directory is capped (e.g. by dfs.namenode.fs-limits.max-directory-items in HDFS).

The directory item limit of /user/${USER}/tmp/staging/${USER} is exceeded: limit=1048576 items=1048576
	at org.apache.hadoop.hdfs.server.namenode.FSDirectory.verifyMaxDirItems(FSDirectory.java:1205)

Issue Links

• links to: GitHub Pull Request #5818