[ARROW-5004] [Python] Confusing behaviour with boolean partition keys

• Type: Bug
• Status: Closed
• Priority: Minor
• Resolution: Duplicate
• Affects Version/s: 0.12.1
• Component/s: Python
• Labels: parquet
• External issue URL: https://github.com/apache/arrow/issues/21503
• Assignee: Unassigned
• Reporter: Scott Taylor
• Created: 25/Mar/19 10:52
• Updated: 11/Jan/23 07:37
• Resolved: 28/Mar/19 22:22

Description

https://github.com/apache/arrow/blob/3129732a18210d0c8921b45f79be4f34eadf0cc3/python/pyarrow/parquet.py#L686

Here the type of a partition key is converted to match the type of a filter variable.

using the write_to_dataset function allows boolean partition keys (True or False) but these silently break at the linked line as bool('False') evaluates as True.

I understand a docstring (https://github.com/apache/arrow/blob/3129732a18210d0c8921b45f79be4f34eadf0cc3/python/pyarrow/parquet.py#L653) refers to only string or int partition variables being supported although this is somewhat buried away from the user facing API.

It may be beneficial to detect the boolean case and raise a warning or to ensure the function returns a more intuitive output when partition key is 'False' and the filter variable is False.