[SPARK-1857] map() with lookup() causes exception

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 0.9.0
• Assignee: Unassigned
• Reporter: Michael Malak
• Created: 15/May/14 22:50
• Updated: 24/Nov/14 04:53
• Resolved: 25/May/14 21:01

Description

Using map() and lookup() in conjunction throws an exception

val a = sc.parallelize(Array(11))
val m = sc.parallelize(Array((11,21)))
a.map(m.lookup(_)(0)).collect

14/05/14 15:03:35 ERROR Executor: Exception in task ID 23
scala.MatchError: null
at org.apache.spark.rdd.PairRDDFunctions.lookup(PairRDDFunctions.scala:551)

A workaround is:

a.map((_,0)).join(m).map(_._2._2).collect