[ARROW-11009] [Python] Add environment variable to elect default usage of system memory allocator instead of jemalloc/mimalloc

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: C++, Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26928
• Assignee: Antoine Pitrou
• Reporter: Wes McKinney
• Created: 22/Dec/20 19:45
• Updated: 11/Jan/23 08:16
• Resolved: 11/Jan/21 14:29

Description

We routinely get reports like ARROW-11007 where there is suspicion of a memory leak (which may or may not be valid) — having an easy way (requiring no changes to application code) to toggle usage of the non-system memory allocator would help with determining whether the memory usage patterns are the result of the allocator being used.

Issue Links

• is related to: ARROW-9974 [Python][C++] pyarrow version 1.0.1 throws Out Of Memory exception while reading large number of files using ParquetDataset
• relates to: ARROW-11007 [Python] Memory leak in pq.read_table and table.to_pandas
• links to: GitHub Pull Request #9105