[ARROW-11279] [Rust][Parquet] ArrowWriter Definition Levels Memory Usage

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: Rust
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27180
• Assignee: R J
• Reporter: R J
• Created: 17/Jan/21 00:35
• Updated: 11/Jan/23 08:18
• Resolved: 20/Jan/21 04:07

Description

In the rust implementation of an Arrow RecordBatch writer to parquet (3.0.0-SNAPSHOT), the ArrowWriter::write call potentially allocates more memory than required.

For a RecordBatch with m rows and n columns, ArrowWriter::write allocates m*n definition levels, leading to m times the required memory usage.

Issue Links

• links to: GitHub Pull Request #9222