[ARROW-5993] [Python] Reading a dictionary column from Parquet results in disproportionate memory usage

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 0.14.0
• Fix Version/s: 0.15.0
• Component/s: Python
• Labels: memory parquet
• External issue URL: https://github.com/apache/arrow/issues/22400
• Assignee: Wes McKinney
• Reporter: Daniel Haviv
• Created: 21/Jul/19 11:44
• Updated: 11/Jan/23 07:44
• Resolved: 20/Aug/19 16:50

Description

I'm using pyarrow to read a 40MB parquet file.

When reading all of the columns besides the "body" columns, the process peaks at 170MB.

Reading only the "body" column results in over 6GB of memory used.

I made the file publicly accessible: s3://dhavivresearch/pyarrow/demofile.parquet

 

 

Issue Links

• duplicates: ARROW-6060 [Python] too large memory cost using pyarrow.parquet.read_table with use_threads=True
• is caused by: ARROW-6060 [Python] too large memory cost using pyarrow.parquet.read_table with use_threads=True
• relates to: ARROW-3772 [C++] Read Parquet dictionary encoded ColumnChunks directly into an Arrow DictionaryArray