XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.6.0
    • Component/s: SQL
    • Labels:
      None
    • Target Version/s:

      Description

      Users of Spark SQL Datasets should be able to use objects that we don't understand (i.e. today not primitives or case classes) in Spark Datasets. To allow this, lets add a default encoder that uses kryo to store the class as bytes and presents it as a single field value.

      For now, lets require the user to explicitly pass the encoder, but as a bonus we can explore the option to use this as a fallback automatically.

      This encoder should be usable in:

      • as
      • map/flatMap, etc
      • Custom Aggregators
      • Scala/Java

        Attachments

          Activity

            People

            • Assignee:
              rxin Reynold Xin
              Reporter:
              rxin Reynold Xin
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: