Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-15989

PySpark SQL python-only UDTs don't support nested types

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 2.0.0
    • 2.0.1, 2.1.0
    • PySpark, SQL
    • None

    Description

      This notebook demonstrates the bug.

      The obvious issue is that nested UDTs are not supported if the UDT is Python-only. Looking at the exception thrown, this seems to be because the encoder on the Java end tries to encode the UDT as a Java class, which doesn't exist for the [[PythonOnlyUDT]].

      Attachments

        Issue Links

          Activity

            People

              viirya L. C. Hsieh
              vlad.feinberg Vladimir Feinberg
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: