Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-13094

No encoder implicits for Seq[Primitive]

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 1.6.0
    • Fix Version/s: 1.6.1, 2.0.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      Dataset aggregators with complex types fail with unable to find encoder for type stored in a Dataset. Though Datasets with these complex types are supported.

      val arraySum = new Aggregator[Seq[Float], Seq[Float],
        Seq[Float]] with Serializable {
        def zero: Seq[Float] = Nil
        // The initial value.
        def reduce(currentSum: Seq[Float], currentRow: Seq[Float]) =
          sumArray(currentSum, currentRow)
        def merge(sum: Seq[Float], row: Seq[Float]) = sumArray(sum, row)
        def finish(b: Seq[Float]) = b // Return the final result.
        def sumArray(a: Seq[Float], b: Seq[Float]): Seq[Float] = {
          (a, b) match {
            case (Nil, Nil) => Nil
            case (Nil, row) => row
            case (sum, Nil) => sum
            case (sum, row) => (a, b).zipped.map { case (a, b) => a + b }
          }
        }
      }.toColumn
      
      <console>:47: error: Unable to find encoder for type stored in a Dataset.  Primitive types (Int, String, etc) and Product types (case classes) are supported by importing sqlContext.implicits._  Support for serializing other types will be added in future releases.
             }.toColumn
      

        Attachments

          Activity

            People

            • Assignee:
              marmbrus Michael Armbrust
              Reporter:
              deenar Deenar Toraskar
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: