[SPARK-42023] createDataFrame should corse types of string false to bool false - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Sub-task
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: 3.4.0
Fix Version/s: 3.4.0
Component/s: Connect
Labels:
None

Description


pyspark/sql/tests/test_types.py:249 (TypesParityTests.test_infer_schema_upcast_boolean_to_string)
self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_infer_schema_upcast_boolean_to_string>

    def test_infer_schema_upcast_boolean_to_string(self):
>       df = self.spark.createDataFrame([[True, 1], ["false", 1]], schema=["a", "b"])

../test_types.py:251: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../connect/session.py:282: in createDataFrame
    _table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in _data])
pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
    ???
pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
    ???
pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
    ???
pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
    ???
pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
    ???
pyarrow/array.pxi:320: in pyarrow.lib.array
    ???
pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Could not convert 'false' with type str: tried to convert to boolean

pyarrow/error.pxi:100: ArrowInvalid