[ARROW-13254] [Python] Processes killed and semaphore objects leaked when reading pandas data

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: 5.0.0
• Component/s: Python
• Environment: OS name and version: macOS 11.4 ; Python version: 3.8.10 ; Pyarrow version: 4.0.1
• External issue URL: https://github.com/apache/arrow/issues/28936
• Assignee: Weston Pace
• Reporter: Koyomi Akaguro
• Created: 02/Jul/21 19:51
• Updated: 11/Jan/23 08:31
• Resolved: 15/Jul/21 13:55

Description

When I run pa.Table.from_pandas(df) for a >1G dataframe, it reports
 
 Killed: 9 ../anaconda3/envs/py38/lib/python3.8/multiprocessing/resource_tracker.py:216: UserWarning: resource_tracker: There appear to be 1 leaked semaphore objects to clean up at shutdown
 
 

Issue Links

• duplicates: ARROW-12983 [C++][Python] Converter::Extend gets stuck in infinite loop causing OOM if values don't fit in single chunk