[SPARK-1199] Type mismatch in Spark shell when using case class defined in shell

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 1.1.0
• Component/s: Spark Core
• Target Version/s: 1.0.1, 1.1.0
• Assignee: Prashant Sharma
• Reporter: Andrew Kerr
• Created: 06/Mar/14 12:51
• Updated: 15/Jun/16 18:08
• Resolved: 04/Jul/14 07:06

Description

NOTE: This issue was fixed in 1.0.1, but the fix was reverted in Spark 1.0.2 pending further testing. The final fix will be in Spark 1.1.0.

Define a class in the shell:

case class TestClass(a:String)

and an RDD

val data = sc.parallelize(Seq("a")).map(TestClass(_))

define a function on it and map over the RDD

def itemFunc(a:TestClass):TestClass = a
data.map(itemFunc)

Error:

<console>:19: error: type mismatch;
 found   : TestClass => TestClass
 required: TestClass => ?
              data.map(itemFunc)

Similarly with a mapPartitions:

def partitionFunc(a:Iterator[TestClass]):Iterator[TestClass] = a
data.mapPartitions(partitionFunc)

<console>:19: error: type mismatch;
 found   : Iterator[TestClass] => Iterator[TestClass]
 required: Iterator[TestClass] => Iterator[?]
Error occurred in an application involving default arguments.
              data.mapPartitions(partitionFunc)

The behavior is the same whether in local mode or on a cluster.

This isn't specific to RDDs. A Scala collection in the Spark shell has the same problem.

scala> Seq(TestClass("foo")).map(itemFunc)
<console>:15: error: type mismatch;
 found   : TestClass => TestClass
 required: TestClass => ?
              Seq(TestClass("foo")).map(itemFunc)
                                        ^

When run in the Scala console (not the Spark shell) there are no type mismatch errors.

Issue Links

• breaks: SPARK-2576 slave node throws NoClassDefFoundError $line11.$read$ when executing a Spark QL query on HDFS CSV file
• is duplicated by: SPARK-1836 REPL $outer type mismatch causes lookup() and equals() problems
• is duplicated by: SPARK-2330 Spark shell has weird scala semantics
• is related to: SPARK-15964 Assignment to RDD-typed val fails
• is superceded by: SPARK-5149 Type mismatch when defining classes in Spark REPL
• relates to: SPARK-2452 Multi-statement input to spark repl does not work
• links to: [Github] Pull Request #1176 (ScrapCodes)
• links to: [Github] Pull Request #1179 (ScrapCodes)