[ARROW-16651] [Python] Casting Table to new schema ignores nullability of fields

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 10.0.0
• Component/s: Python
• Labels: good-first-issue good-second-issue kernel pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/32000
• Assignee: Kshiteej K
• Reporter: Joris Van den Bossche
• Created: 25/May/22 08:56
• Updated: 11/Jan/23 11:45
• Resolved: 08/Sep/22 12:34

Description

Similar to ARROW-15478, but not for nested fields but just for casting a full Table (in theory that could be the same code, but currently the Table.cast logic is implemented in cython).

So currently when casting a Table to a new schema, the nullability of the fields in the schema is ignored (and as a result you get an "invalid" schema indicating a field is non-nullable that actually can have nulls):

>>> table = pa.table({'a': [None, 1]})
>>> table
pyarrow.Table
a: int64
----
a: [[null,1]]

>>> new_schema = pa.schema([pa.field("a", "int64", nullable=False)])
>>> table.cast(new_schema)
pyarrow.Table
a: int64 not null
----
a: [[null,1]]

Issue Links

• is related to: ARROW-15478 [C++] Creating (or casting to) list array with non-nullable field doesn't check nulls
• links to: GitHub Pull Request #14048