[SPARK-2498] Concurrent initialization of various DataType objects causes exceptions

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.0.1
• Fix Version/s: 1.0.2, 1.1.0
• Component/s: SQL
• Target Version/s: 1.0.2, 1.1.0
• Assignee: Zongheng Yang
• Reporter: Zongheng Yang
• Created: 15/Jul/14 23:28
• Updated: 16/Jul/14 00:59
• Resolved: 16/Jul/14 00:59

Description

Reported by Keith Simmons and gorenuru here. This is probably due to the fact that Scala Reflection API is not thread-safe for 2.10.* versions; see SI-6240 for more details.

The purposed fix is to add a lock object that is synchronized on when various data type objects make use of the reflection API (during their initializations or other places in program execution). (Longer-term, we could consider evolving this to a SparkSQL-global lock, and all calls to the reflection APi from Spark SQL should probably synchronize on it.)