[ARROW-1858] [Python] Add documentation about parquet.write_to_dataset and related methods

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.10.0
• Component/s: Python
• Labels: beginner pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/17851
• Assignee: Donal Simmie
• Reporter: Wes McKinney
• Created: 25/Nov/17 22:43
• Updated: 11/Jan/23 07:17
• Resolved: 21/Apr/18 20:22

Description

pyarrow does not only allow one to write to a single Parquet file but you can also write only the schema metadata for a full multi-file dataset. This dataset can also be automatically partitioned by one or more columns. At the moment, this functionality is not really visible in the documentation. You mainly find the API documentation for it but we should have a small tutorial-like section that explains the differences and use cases for each of these functions.

See also https://stackoverflow.com/questions/47482434/can-pyarrow-write-multiple-parquet-files-to-a-folder-like-fastparquets-file-sch

Issue Links

• links to: GitHub Pull Request #1925