Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-18891

Support for specific collection types

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 1.6.3, 2.1.0
    • Fix Version/s: 2.3.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      Encoders treat all collections the same (i.e. Seq vs List) which force users to only define classes with the most generic type.

      An example error:

      case class SpecificCollection(aList: List[Int])
      Seq(SpecificCollection(1 :: Nil)).toDS().collect()
      
      java.lang.RuntimeException: Error while decoding: java.util.concurrent.ExecutionException: java.lang.Exception: failed to compile: org.codehaus.commons.compiler.CompileException: File 'generated.java', Line 98, Column 120: No applicable constructor/method found for actual parameters "scala.collection.Seq"; candidates are: "line29e7e4b1e36445baa3505b2e102aa86b29.$read$$iw$$iw$$iw$$iw$SpecificCollection(scala.collection.immutable.List)"
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                michalsenkyr Michal Šenkýř
                Reporter:
                marmbrus Michael Armbrust
              • Votes:
                7 Vote for this issue
                Watchers:
                19 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: