[ARROW-2628] [Python] parquet.write_to_dataset is memory-hungry on large DataFrames

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: C++, Python
• Labels: dataset dataset-parquet-write parquet
• External issue URL: https://github.com/apache/arrow/issues/19025
• Assignee: Unassigned
• Reporter: Wes McKinney
• Created: 22/May/18 00:39
• Updated: 11/Jan/23 07:22

Description

See discussion in https://github.com/apache/arrow/issues/1749. We should consider strategies for writing very large tables to a partitioned directory scheme.

Issue Links

• duplicates: ARROW-2709 [Python] write_to_dataset poor performance when splitting