[SPARK-42020] createDataFrame with UDT

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.1
• Component/s: Connect
• Assignee: Takuya Ueshin
• Reporter: Hyukjin Kwon
• Created: 12/Jan/23 05:52
• Updated: 20/Mar/23 19:09
• Resolved: 20/Mar/23 00:24

Description

pyspark/sql/tests/test_types.py:596 (TypesParityTests.test_apply_schema_with_udt)
self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_apply_schema_with_udt>

    def test_apply_schema_with_udt(self):
        row = (1.0, ExamplePoint(1.0, 2.0))
        schema = StructType(
            [
                StructField("label", DoubleType(), False),
                StructField("point", ExamplePointUDT(), False),
            ]
        )
>       df = self.spark.createDataFrame([row], schema)

../test_types.py:605: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../connect/session.py:282: in createDataFrame
    _table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in _data])
pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
    ???
pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
    ???
pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
    ???
pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
    ???
pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
    ???
pyarrow/array.pxi:320: in pyarrow.lib.array
    ???
pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Could not convert ExamplePoint(1.0,2.0) with type ExamplePoint: did not recognize Python value type when inferring an Arrow data type

pyarrow/error.pxi:100: ArrowInvalid

Issue Links

• links to: [Github] Pull Request #40402 (ueshin)