[HADOOP-15469] S3A directory committer commit job fails if _temporary directory created under dest

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.1
• Component/s: fs/s3

Environment

spark test runs

• Target Version/s: 3.1.1
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 15/May/18 14:44
• Updated: 10/Jun/19 13:01
• Resolved: 17/May/18 18:09

Description

The directory staging committer fails in commit job if any temporary files/dirs have been created. Spark work can create such a dir for placement of absolute files.

This is because commitJob() looks for the dest dir existing, not containing non-hidden files.
As the comment says, "its kind of superfluous". More specifically, it means jobs which would commit with the classic committer & overwrite=false will fail

Proposed fix: remove the check