[SPARK-534] Make SparkContext thread-safe

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Invalid
• Affects Version/s: 0.5.0, 0.5.1, 0.5.2, 0.6.0, 0.6.1, 0.6.2, 0.7.0, 0.7.1, 0.7.2, 0.7.3
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: tjhunter (Inactive)
• Created: 15/Sep/12 09:49
• Updated: 18/Oct/14 21:41
• Resolved: 18/Oct/14 21:40

Description

SparkEnv (used by SparkContext) is not thread-safe and it causes issues with scala's Futures and parrallel collections.
For example, this will not work:

val f = Futures.future({
  sc.textFile("hdfs://....")
})
f.apply()

Workaround for now:

val f = Futures.future({
  SparkEnv.set(sc.env)
  sc.textFile("hdfs://....")
})
f.apply()