[ARROW-4120] [Python] Define process for testing procedures that check for no macro-level memory leaks

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.17.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/15919
• Assignee: Wes McKinney
• Reporter: Wes McKinney
• Created: 26/Dec/18 22:37
• Updated: 11/Jan/23 07:31
• Resolved: 09/Mar/20 22:49

Description

Some kinds of memory leaks may be difficult to unit test for, and they may not cause valgrind errors necessarily

I had written some ad hoc leak tests in https://github.com/apache/arrow/blob/master/python/scripts/test_leak.py. We have some more of this in ARROW-3324.

It would be useful to be able to create a sort of "test suite" of memory leak checks. They are a bit too intensive to run in CI (since you may have to run something many iterations to see whether it leaks), but we could run them in a nightly build

Issue Links

• is related to: ARROW-7956 [Python] Memory leak in pyarrow functions .ipc.serialize_pandas/deserialize_pandas
• is related to: ARROW-8048 [Python] Run memory leak tests nightly as follow up to ARROW-4120
• links to: GitHub Pull Request #6551