[ARROW-5863] [Python] Segmentation Fault via pytest-runner

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.14.0
• Fix Version/s: 0.14.1, 0.15.0
• Component/s: Python
• Labels: pull-request-available

Environment

$ uname -a
Linux aleph 5.1.15-arch1-1-ARCH #1 SMP PREEMPT Tue Jun 25 04:49:39 UTC 2019 x86_64 GNU/Linux

$ python --version
Python 3.7.3

$ pip freeze | grep -P "(pyarrow|pytest)"
pyarrow==0.14.0
pytest==5.0.0
pytest-benchmark==3.2.2
pytest-cov==2.7.1
pytest-env==0.6.2
pytest-forked==1.0.2
pytest-html==1.21.1
pytest-metadata==1.8.0
pytest-mock==1.10.4
pytest-runner==5.1
pytest-sugar==0.9.2
pytest-xdist==1.29.0

• External issue URL: https://github.com/apache/arrow/issues/22279
• Assignee: Wes McKinney
• Reporter: Josh Bode
• Created: 05/Jul/19 13:26
• Updated: 11/Jan/23 07:43
• Resolved: 08/Jul/19 21:28

Description

When running pytest on projects using pyarrow==0.14.0 on Linux, I am getting segmentation faults, but interestingly only when run via pytest-runner (which provides the setup.py pytest command)

This works (i.e. pytest directly):

$ pytest

Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0, pytest-sugar 0.9.2)
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/josh/scratch/pyarrow-issue
plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4, xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1, html-1.21.1, benchmark-3.2.2, metadata-1.8.0
collecting ...
tests/test_pyarrow.py ✓ 100% ██████████

Results (0.09s):
1 passed

However, this does not work, ending in a segmentation fault, even though the tests pass:

$ python setup.py pytest

running pytest
running egg_info
writing pyarrow_issue.egg-info/PKG-INFO
writing dependency_links to pyarrow_issue.egg-info/dependency_links.txt
writing requirements to pyarrow_issue.egg-info/requires.txt
writing top-level names to pyarrow_issue.egg-info/top_level.txt
reading manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
writing manifest file 'pyarrow_issue.egg-info/SOURCES.txt'
running build_ext

Test session starts (platform: linux, Python 3.7.3, pytest 5.0.0, pytest-sugar 0.9.2)
benchmark: 3.2.2 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=0.000005 max_time=1.0 calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/josh/scratch/pyarrow-issue
plugins: sugar-0.9.2, Flask-Dance-2.2.0, env-0.6.2, mock-1.10.4, xdist-1.29.0, requests-mock-1.6.0, forked-1.0.2, dash-1.0.0, cov-2.7.1, html-1.21.1, benchmark-3.2.2, metadata-1.8.0
collecting ...
tests/test_pyarrow.py ✓ 100% ██████████

Results (0.07s):
1 passed
zsh: segmentation fault (core dumped) python setup.py pytest

backtrace from gdb

Thread 1 "python" received signal SIGSEGV, Segmentation fault.
0x00007ffff7c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0

(gdb) bt
#0 0x00007ffff7c10b58 in ?? () from /usr/lib/libpython3.7m.so.1.0
#1 0x00007ffff7ae46cc in ?? () from /usr/lib/libpython3.7m.so.1.0
#2 0x00007ffff023a6b3 in arrow::py::PyExtensionType::~PyExtensionType() ()
from /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow_python.so.14
#3 0x00007fffed5e6467 in std::unordered_map<std::string, std::shared_ptr<arrow::ExtensionType>, std::hash<std::string>, std::equal_to<std::string>, std::allocator<std::pair<std::string const, std::shared_ptr<arrow::ExtensionType> > > >::~unordered_map() ()
from /home/josh/.virtualenvs/default/lib/python3.7/site-packages/pyarrow/./libarrow.so.14
#4 0x00007ffff7de5e70 in __run_exit_handlers () from /usr/lib/libc.so.6
#5 0x00007ffff7de5fae in exit () from /usr/lib/libc.so.6
#6 0x00007ffff7dcfeea in __libc_start_main () from /usr/lib/libc.so.6
#7 0x000055555555505e in _start ()

I have observed this behaviour on my machine running natively, and also via docker. Also, 0.13.0 does not exhibit this behaviour

 

Issue Links

• links to: GitHub Pull Request #4824