The directory staging committer fails in commit job if any temporary files/dirs have been created. Spark work can create such a dir for placement of absolute files.
This is because commitJob() looks for the dest dir existing, not containing non-hidden files.
As the comment says, "its kind of superfluous". More specifically, it means jobs which would commit with the classic committer & overwrite=false will fail
Proposed fix: remove the check