[SPARK-12087] DStream.saveAsHadoopFiles can throw ConcurrentModificationException

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.1, 1.4.1, 1.5.2
• Fix Version/s: 1.4.2, 1.5.3, 1.6.0
• Component/s: DStreams
• Target Version/s: 1.6.0
• Assignee: Tathagata Das
• Reporter: Tathagata Das
• Created: 02/Dec/15 02:57
• Updated: 02/Dec/15 05:07
• Resolved: 02/Dec/15 05:07

Description

The JobConf object created in DStream.saveAsHadoopFiles is used concurrently in multiple places:

• The JobConf is updated by RDD.saveAsHadoopFile() before the job is launched
• The JobConf is serialized as part of the DStream checkpoints.

These concurrent accesses (updating in one thread, while the another thread is serializing it) can lead to concurrentModidicationException in the underlying Java hashmap using in the internal Hadoop Configuration object.

Issue Links

• links to: [Github] Pull Request #10088 (tdas)