[ARROW-2057] [Python] Configure size of data pages in pyarrow.parquet.write_table

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.14.0
• Component/s: Python
• Labels: beginner parquet pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18035
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 30/Jan/18 18:40
• Updated: 11/Jan/23 07:18
• Resolved: 18/Jun/19 17:18

Description

It would be useful to be able to set the size of data pages (within Parquet column chunks) from Python. The current default is set to 1MiB at https://github.com/apache/parquet-cpp/blob/0875e43010af485e1c0b506d77d7e0edc80c66cc/src/parquet/properties.h#L81. It might be useful in some situations to lower this for more granular access.

We should provide this value as a parameter to pyarrow.parquet.write_table.

Issue Links

• links to: GitHub Pull Request #4597