Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33489

Support null for conversion from and to Arrow type

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 3.0.1
    • 3.2.0
    • PySpark
    • None

    Description

      I got below error when using from_arrow_type() in pyspark.sql.pandas.types

      Unsupported type in conversion from Arrow: null

      I noticed NullType exists under pyspark.sql.types so it seems possible to convert from pyarrow null to pyspark null type and vice versa.

      Attachments

        Issue Links

          Activity

            People

              ueshin Takuya Ueshin
              cactice Yuya Kanai
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: