Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-1199

Type mismatch in Spark shell when using case class defined in shell

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Fixed
    • 0.9.0
    • 1.1.0
    • Spark Core
    • None

    Description

      NOTE: This issue was fixed in 1.0.1, but the fix was reverted in Spark 1.0.2 pending further testing. The final fix will be in Spark 1.1.0.

      Define a class in the shell:

      case class TestClass(a:String)
      

      and an RDD

      val data = sc.parallelize(Seq("a")).map(TestClass(_))
      

      define a function on it and map over the RDD

      def itemFunc(a:TestClass):TestClass = a
      data.map(itemFunc)
      

      Error:

      <console>:19: error: type mismatch;
       found   : TestClass => TestClass
       required: TestClass => ?
                    data.map(itemFunc)
      

      Similarly with a mapPartitions:

      def partitionFunc(a:Iterator[TestClass]):Iterator[TestClass] = a
      data.mapPartitions(partitionFunc)
      
      <console>:19: error: type mismatch;
       found   : Iterator[TestClass] => Iterator[TestClass]
       required: Iterator[TestClass] => Iterator[?]
      Error occurred in an application involving default arguments.
                    data.mapPartitions(partitionFunc)
      

      The behavior is the same whether in local mode or on a cluster.

      This isn't specific to RDDs. A Scala collection in the Spark shell has the same problem.

      scala> Seq(TestClass("foo")).map(itemFunc)
      <console>:15: error: type mismatch;
       found   : TestClass => TestClass
       required: TestClass => ?
                    Seq(TestClass("foo")).map(itemFunc)
                                              ^
      

      When run in the Scala console (not the Spark shell) there are no type mismatch errors.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            prashant Prashant Sharma
            andrewkerr Andrew Kerr
            Votes:
            1 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment