[ARROW-1983] [Python] Add ability to write parquet `_metadata` file

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: C++, Python
• Labels: beginner parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17967
• Assignee: Rick Zamora
• Reporter: Jim Crist
• Created: 10/Jan/18 20:48
• Updated: 11/Jan/23 07:18
• Resolved: 06/Jun/19 21:38

Description

Currently pyarrow.parquet can only write the _common_metadata file (mostly just schema information). It would be useful to add the ability to write a _metadata file as well. This should include information about each row group in the dataset, including summary statistics. Having this summary file would allow filtering of row groups without needing to access each file beforehand.

This would require that the user is able to get the written RowGroups out of a pyarrow.parquet.write_table call and then give these objects as a list to new function that then passes them on as C++ objects to parquet-cpp that generates the respective _metadata file.

Issue Links

• blocks: ARROW-2801 [Python][C++][Dataset] Implement split_row_groups for ParquetDataset
• is related to: ARROW-5258 [C++/Python] Expose file metadata of dataset pieces to caller
• relates to: ARROW-13269 [C++] [Dataset] Improve the _metadata example to show how to properly create _metadata if there is a partitioning
• relates to: ARROW-5349 [Python/C++] Provide a way to specify the file path in parquet ColumnChunkMetaData
• links to: GitHub Pull Request #4166
• links to: GitHub Pull Request #4236
• links to: GitHub Pull Request #4405