[ARROW-6780] [C++][Parquet] Support DurationType in writing/reading parquet

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 8.0.0
• Component/s: C++
• Labels: parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/23117
• Assignee: Joris Van den Bossche
• Reporter: Joris Van den Bossche
• Created: 03/Oct/19 13:51
• Updated: 11/Jan/23 07:49
• Resolved: 17/Feb/22 15:30

Description

Currently this is not supported:

In [37]: table = pa.table({'a': pa.array([1, 2], pa.duration('s'))}) 

In [39]: table
Out[39]: 
pyarrow.Table
a: duration[s]

In [41]: pq.write_table(table, 'test_duration.parquet')
...
ArrowNotImplementedError: Unhandled type for Arrow to Parquet schema conversion: duration[s]

There is no direct mapping to Parquet logical types. There is an INTERVAL type, but this more matches Arrow's ( YEAR_MONTH or DAY_TIME) interval type.

But, those duration values could be stored as just integers, and based on the serialized arrow schema, it could be restored when reading back in.

Issue Links

• links to: GitHub Pull Request #12449