Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-19741

ClassCastException when using Dataset with type containing value types

Attach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Incomplete
    • 2.1.0
    • None
    • Spark Core, SQL

    • JDK 8 on Ubuntu
      Scala 2.11.8
      Spark 2.1.0

    Description

      The following code reproduces the error

      final case class Foo(id: Int) extends AnyVal
final case class Bar(foo: Foo)
val foo = Foo(5)
val bar = Bar(foo)
import spark.implicits._
spark.sparkContext.parallelize(0 to 10).toDS().map(_ => bar).collect()

      Attachments

        Issue Links

        relates to

        Improvement - An improvement or enhancement to an existing feature or task. SPARK-20384 supporting value classes over primitives in DataSets

        • Minor - Minor loss of function, or other problem where easy workaround is present.
        • Resolved

        Activity
          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            lioron Lior Regev
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment