Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-17414

Set type is not supported for creating data frames

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Closed
    • Minor
    • Resolution: Fixed
    • 2.0.0
    • 2.3.0
    • SQL
    • None

    Description

      For a case class that has a field of type Set, createDataFrame() method throws an exception saying "Schema for type Set is not supported". Exception is raised by the org.apache.spark.sql.catalyst.ScalaReflection class where Array, Seq and Map types are supported but Set is not. It would be nice to support Set here by default instead of having to write a custom Encoder.

      Attachments

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. SPARK-21204 RuntimeException with Set and Case Class in Spark 2.1.1

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              Unassigned Unassigned
              emre.colak Emre Colak
              Votes:
              1 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: