[ARROW-15234] [Python] Possible crash with custom CSV invalid row handler

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 7.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/30732
• Assignee: Antoine Pitrou
• Reporter: Antoine Pitrou
• Created: 03/Jan/22 15:10
• Updated: 11/Jan/23 08:45
• Resolved: 03/Jan/22 16:57

Description

The crash happens on handler finalization and can be reproduced reliable here:

$ taskset -c 1 python -m pytest -r s --tb=native pyarrow/tests/test_csv.py::TestThreadedCSVTableRead::test_invalid_row_handler
=========================================================================== test session starts ===========================================================================
platform linux -- Python 3.9.7, pytest-6.2.5, py-1.10.0, pluggy-1.0.0
rootdir: /home/antoine/arrow/dev/python, configfile: setup.cfg
plugins: repeat-0.9.1, lazy-fixture-0.6.3, hypothesis-6.23.2
collected 1 item                                                                                                                                                          

pyarrow/tests/test_csv.py Fatal Python error: Segmentation fault

Thread 0x00007fd0780af740 (most recent call first):
  File "/home/antoine/arrow/dev/python/pyarrow/tests/test_csv.py", line 670 in read_csv
  File "/home/antoine/arrow/dev/python/pyarrow/tests/test_csv.py", line 675 in read_bytes
  File "/home/antoine/arrow/dev/python/pyarrow/tests/test_csv.py", line 652 in test_invalid_row_handler
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/python.py", line 183 in pytest_pyfunc_call
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/python.py", line 1641 in runtest
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/runner.py", line 162 in pytest_runtest_call
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/runner.py", line 255 in <lambda>
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/runner.py", line 311 in from_call
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/runner.py", line 254 in call_runtest_hook
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/runner.py", line 215 in call_and_report
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/runner.py", line 126 in runtestprotocol
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/runner.py", line 109 in pytest_runtest_protocol
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/main.py", line 348 in pytest_runtestloop
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/main.py", line 323 in _main
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/main.py", line 269 in wrap_session
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/main.py", line 316 in pytest_cmdline_main
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_callers.py", line 39 in _multicall
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_manager.py", line 80 in _hookexec
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pluggy/_hooks.py", line 265 in __call__
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/config/__init__.py", line 162 in main
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/_pytest/config/__init__.py", line 185 in console_main
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/site-packages/pytest/__main__.py", line 5 in <module>
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/runpy.py", line 87 in _run_code
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.9/runpy.py", line 197 in _run_module_as_main
Erreur de segmentation (core dumped)

Issue Links

• is caused by: ARROW-15116 [Python] Expose invalid_row_handler for CSV reader
• links to: GitHub Pull Request #12063