[ARROW-10145] [C++][Dataset] Assert integer overflow in partitioning falls back to string

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: C++
• Labels: dataset pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26156
• Assignee: Ben Kietzman
• Reporter: Joris Van den Bossche
• Created: 30/Sep/20 15:58
• Updated: 11/Jan/23 08:11
• Resolved: 15/Oct/20 19:29

Description

From https://stackoverflow.com/questions/64137664/how-to-override-type-inference-for-partition-columns-in-hive-partitioned-dataset

Small reproducer:

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({'part': [3760212050]*10, 'col': range(10)})
pq.write_to_dataset(table, "test_int64_partition", partition_cols=['part'])

In [35]: pq.read_table("test_int64_partition/")
...
ArrowInvalid: error parsing '3760212050' as scalar of type int32
In ../src/arrow/scalar.cc, line 333, code: VisitTypeInline(*type_, this)
In ../src/arrow/dataset/partition.cc, line 218, code: (_error_or_value26).status()
In ../src/arrow/dataset/partition.cc, line 229, code: (_error_or_value27).status()
In ../src/arrow/dataset/discovery.cc, line 256, code: (_error_or_value17).status()

In [36]: pq.read_table("test_int64_partition/", use_legacy_dataset=True)
Out[36]: 
pyarrow.Table
col: int64
part: dictionary<values=int64, indices=int32, ordered=0>

Issue Links

• links to: GitHub Pull Request #8462