[ARROW-4470] [Python] Pyarrow using considerable more memory when reading partitioned Parquet file

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 0.12.0
• Fix Version/s: 0.16.0
• Component/s: Python
• Labels: dataset datasets parquet
• External issue URL: https://github.com/apache/arrow/issues/21027
• Assignee: Unassigned
• Reporter: Ivan SPM
• Created: 04/Feb/19 10:54
• Updated: 11/Jan/23 07:34
• Resolved: 21/Aug/19 23:21

Description

Hi,

I have a partitioned Parquet table in Impala in HDFS, using Hive metastore, with the following structure:

/data/myparquettable/year=2016/data/myparquettable/year=2016/myfile_1.prt

/data/myparquettable/year=2016/myfile_2.prt

/data/myparquettable/year=2016/myfile_3.prt

/data/myparquettable/year=2017

/data/myparquettable/year=2017/myfile_1.prt

/data/myparquettable/year=2017/myfile_2.prt

/data/myparquettable/year=2017/myfile_3.prt

and so on. I need to work with one partition, so I copied one partition to a local filesystem:

hdfs fs -get /data/myparquettable/year=2017 /local/

so now I have some data on the local disk:

{{/local/year=2017/myfile_1.prt }}{{/local/year=2017/myfile_2.prt }}

etc.I tried to read it using Pyarrow:

import pyarrow.parquet as pqpq.read_parquet('/local/year=2017')

and it starts reading. The problem is that the local Parquet files are around 15GB total, and I blew up my machine memory a couple of times because when reading these files, Pyarrow is using more than 60GB of RAM, and I'm not sure how much it will take because it never finishes. Is this expected? Is there a workaround?