[SPARK-42023] createDataFrame should corse types of string false to bool false

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: Connect
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 12/Jan/23 06:04
• Updated: 31/Jan/23 06:28
• Resolved: 31/Jan/23 06:28

Description

pyspark/sql/tests/test_types.py:249 (TypesParityTests.test_infer_schema_upcast_boolean_to_string)
self = <pyspark.sql.tests.connect.test_parity_types.TypesParityTests testMethod=test_infer_schema_upcast_boolean_to_string>

    def test_infer_schema_upcast_boolean_to_string(self):
>       df = self.spark.createDataFrame([[True, 1], ["false", 1]], schema=["a", "b"])

../test_types.py:251: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../connect/session.py:282: in createDataFrame
    _table = pa.Table.from_pylist([dict(zip(_cols, list(item))) for item in _data])
pyarrow/table.pxi:3700: in pyarrow.lib.Table.from_pylist
    ???
pyarrow/table.pxi:5221: in pyarrow.lib._from_pylist
    ???
pyarrow/table.pxi:3575: in pyarrow.lib.Table.from_arrays
    ???
pyarrow/table.pxi:1383: in pyarrow.lib._sanitize_arrays
    ???
pyarrow/table.pxi:1364: in pyarrow.lib._schema_from_arrays
    ???
pyarrow/array.pxi:320: in pyarrow.lib.array
    ???
pyarrow/array.pxi:39: in pyarrow.lib._sequence_to_array
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

>   ???
E   pyarrow.lib.ArrowInvalid: Could not convert 'false' with type str: tried to convert to boolean

pyarrow/error.pxi:100: ArrowInvalid

Issue Links

• links to: [Github] Pull Request #39818 (zhengruifeng)