[ARROW-5349] [Python/C++] Provide a way to specify the file path in parquet ColumnChunkMetaData

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: C++, Python
• Labels: parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/21808
• Assignee: Wes McKinney
• Reporter: Joris Van den Bossche
• Created: 16/May/19 13:51
• Updated: 11/Jan/23 07:39
• Resolved: 26/May/19 12:22

Description

After ARROW-5258 / https://github.com/apache/arrow/pull/4236 it is now possible to collect the file metadata while writing different files (then how to write those metadata was not yet addressed -> original issue ARROW-1983).

However, currently, the file_path information in the ColumnChunkMetaData object is not set. This is, I think, expected / correct for the metadata as included within the single file; but for using the metadata in the combined dataset `_metadata`, it needs a file path set.

So if you want to use this metadata for a partitioned dataset, there needs to be a way to specify this file path.
Ideas I am thinking of currently: either, we could specify a file path to be used when writing, or expose the `set_file_path` method on the Python side so you can create an updated version of the metadata after collecting it.

cc pearu mdurant

Issue Links

• is related to: ARROW-1983 [Python] Add ability to write parquet `_metadata` file
• links to: GitHub Pull Request #4361
• links to: GitHub Pull Request #4386