[ARROW-9952] [Python] Use pyarrow.dataset writing for pq.write_to_dataset

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/25979
• Assignee: Joris Van den Bossche
• Reporter: Joris Van den Bossche
• Created: 09/Sep/20 13:05
• Updated: 11/Jan/23 08:10
• Resolved: 10/Oct/20 11:22

Description

Now ARROW-9658 and ARROW-9893 are in, we can explore using the pyarrow.dataset writing capabilities in parquet.write_to_dataset.

Similarly as was done in pq.read_table, we could initially have a keyword to switch between both implementations, eventually defaulting to the new datasets one, and to deprecated the old (inefficient) python implementation.

Issue Links

• links to: GitHub Pull Request #8412