Details
Major Description
SPARK-8029 modified shuffle writers to first stage their data to a temporary file in the same directory as the final destination file and then to atomically rename the file at the end of the write job. However, this change introduced the potential for the temporary output file to be leaked if an exception occurs during the write because the shuffle writers' existing error cleanup code doesn't handle this new temp file.
This is easy to fix: we just need to add a finally block to ensure that the temporary file is guaranteed to be either moved or deleted before existing the shuffle write method.
