[ARROW-6060] [Python] too large memory cost using pyarrow.parquet.read_table with use_threads=True

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.1
• Fix Version/s: 0.15.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/22462
• Assignee: Ben Kietzman
• Reporter: Kun Liu
• Created: 29/Jul/19 11:49
• Updated: 11/Jan/23 07:44
• Resolved: 07/Aug/19 10:31

Description

 I tried to load a parquet file of about 1.8Gb using the following code. It crashed due to out of memory issue.

import pyarrow.parquet as pq
pq.read_table('/tmp/test.parquet')

 However, it worked well with use_threads=True as follows

pq.read_table('/tmp/test.parquet', use_threads=False)

If pyarrow is downgraded to 0.12.1, there is no such problem.

Issue Links

• causes: ARROW-5993 [Python] Reading a dictionary column from Parquet results in disproportionate memory usage
• is duplicated by: ARROW-5993 [Python] Reading a dictionary column from Parquet results in disproportionate memory usage
• is duplicated by: ARROW-6380 Method pyarrow.parquet.read_table has memory spikes from version 0.14
• is related to: ARROW-6230 [R] Reading in Parquet files are 20x slower than reading fst files in R
• relates to: ARROW-6059 [Python] Regression memory issue when calling pandas.read_parquet
• links to: GitHub Pull Request #5016