[SPARK-20038] FileFormatWriter.ExecuteWriteTask.releaseResources() implementations to be re-entrant

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.2.0
• Component/s: Spark Core
• Assignee: Steve Loughran
• Reporter: Steve Loughran
• Created: 20/Mar/17 19:30
• Updated: 16/Apr/17 07:27
• Resolved: 13/Apr/17 20:31

Description

Both FileFormatWriter.ExecuteWriteTask.releaseResources() implementations close() any non-null currentWriter, then set the field to null

However, if the close() call throws an exception in the execution of {FileFormatWriter.executeTask, the exception handler will attempt to cleanup, by calling releaseResources() again. Looking at the codepath, this may cause committer.abortTask() to get skipped on failure.

This surfaces in SPARK-10109 and I've just seen it in HADOOP-14204); Parquet seems to be in the trace as it NPEs the second time it's close() method is called.

Fix: always set currentWriter to null,

Issue Links

• is duplicated by: SPARK-10109 NPE when saving Parquet To HDFS
• relates to: HADOOP-14204 S3A multipart commit failing, "UnsupportedOperationException at java.util.Collections$UnmodifiableList.sort"
• relates to: SPARK-10109 NPE when saving Parquet To HDFS
• links to: [Github] Pull Request #17364 (steveloughran)