[SPARK-9621] Closure inside RDD doesn't properly close over environment

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 1.4.1

Environment

Ubuntu 15.04, spark-1.4.1-bin-hadoop2.6 package

• Assignee: Unassigned
• Reporter: Joe Near
• Created: 04/Aug/15 23:17
• Updated: 14/Apr/20 22:08
• Resolved: 05/Aug/15 07:52

Description

I expect the following:

case class MyTest(i: Int)
val tv = MyTest(1)
val res = sc.parallelize(Array((t: MyTest) => t == tv)).first()(tv)

to be "true." It is "false," when I type this into spark-shell. It seems the closure is changed somehow when it's serialized and deserialized.

Issue Links

• duplicates: SPARK-2620 case class cannot be used as key for reduce