Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1199

Type mismatch in Spark shell when using case class defined in shell

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: 0.9.0
    • Fix Version/s: 1.1.0
    • Component/s: Spark Core
    • Labels:
      None

      Description

      NOTE: This issue was fixed in 1.0.1, but the fix was reverted in Spark 1.0.2 pending further testing. The final fix will be in Spark 1.1.0.

      Define a class in the shell:

      case class TestClass(a:String)
      

      and an RDD

      val data = sc.parallelize(Seq("a")).map(TestClass(_))
      

      define a function on it and map over the RDD

      def itemFunc(a:TestClass):TestClass = a
      data.map(itemFunc)
      

      Error:

      <console>:19: error: type mismatch;
       found   : TestClass => TestClass
       required: TestClass => ?
                    data.map(itemFunc)
      

      Similarly with a mapPartitions:

      def partitionFunc(a:Iterator[TestClass]):Iterator[TestClass] = a
      data.mapPartitions(partitionFunc)
      
      <console>:19: error: type mismatch;
       found   : Iterator[TestClass] => Iterator[TestClass]
       required: Iterator[TestClass] => Iterator[?]
      Error occurred in an application involving default arguments.
                    data.mapPartitions(partitionFunc)
      

      The behavior is the same whether in local mode or on a cluster.

      This isn't specific to RDDs. A Scala collection in the Spark shell has the same problem.

      scala> Seq(TestClass("foo")).map(itemFunc)
      <console>:15: error: type mismatch;
       found   : TestClass => TestClass
       required: TestClass => ?
                    Seq(TestClass("foo")).map(itemFunc)
                                              ^
      

      When run in the Scala console (not the Spark shell) there are no type mismatch errors.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                prashant_ Prashant Sharma
                Reporter:
                andrewkerr Andrew Kerr
              • Votes:
                1 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: