[ARROW-8284] [C++][Dataset] Schema evolution for timestamp columns

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: C++
• Labels: dataset
• External issue URL: https://github.com/apache/arrow/issues/24478
• Assignee: Unassigned
• Reporter: Uwe Korn
• Created: 31/Mar/20 12:47
• Updated: 11/Jan/23 07:59
• Resolved: 15/Apr/21 12:30

Description

In a dataset, one can have timestamp columns with different resolutions. There should be an optional to cast all timestamps to the type mentioned in the schema. A typical example could be that we store a pandas DataFrame with ns precision to Parquet files that only support us resolution in their most widespread from. Then the dataset schema and the actual file content don't match anymore.

Issue Links

• is related to: ARROW-11003 [C++][Dataset] Schema evolution in Dataset scanning