[ARROW-6976] Possible memory leak in pyarrow read_parquet

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 0.15.0
• Component/s: Python
• Environment: linux ubuntu 18.04
• External issue URL: https://github.com/apache/arrow/issues/23291
• Assignee: Unassigned
• Reporter: david cottrell
• Created: 23/Oct/19 15:22
• Updated: 11/Jan/23 07:50
• Resolved: 25/Oct/19 09:05

Description

 

Version and repro info in the gist below.

Not sure if I'm not understanding something from this https://arrow.apache.org/blog/2019/02/05/python-string-memory-0.12/

but there seems to be memory accumulation when that is exacerbated with higher arity objects like strings and dates (not datetimes).

 

I was not able to reproduce the issue on MacOS. Downgrading to 0.14.1 seemed to "fix" or lessen the problem.

 

https://gist.github.com/cottrell/a3f95aa59408d87f925ec606d8783e62

 

Let me know if this post should go elsewhere.

 

 

 

Issue Links

• is duplicated by: ARROW-6874 [Python] Memory leak in Table.to_pandas() when conversion to object dtype