[ARROW-11596] [Python][Dataset] SIGSEGV when executing scan tasks with Python executors

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0
• Component/s: Python
• Labels: dataset datasets pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/27465
• Assignee: David Li
• Reporter: David Li
• Created: 11/Feb/21 16:28
• Updated: 11/Jan/23 08:20
• Resolved: 25/Mar/21 12:55

Description

This crashes for me with a segfault:

import concurrent.futures
import queue

import numpy as np
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.fs as fs
import pyarrow.parquet as pq


schema = pa.schema([("foo", pa.float64())])
table = pa.table([np.random.uniform(size=1024)], schema=schema)
path = "/tmp/foo.parquet"
pq.write_table(table, path)
dataset = pa.dataset.FileSystemDataset.from_paths(
    [path],
    schema=schema,
    format=ds.ParquetFileFormat(),
    filesystem=fs.LocalFileSystem(),
)

with concurrent.futures.ThreadPoolExecutor(2) as executor:
    tasks = dataset.scan()
    q = queue.Queue()

    def _prebuffer():
        for task in tasks:
            iterator = task.execute()
            next(iterator)
            q.put(iterator)

    executor.submit(_prebuffer).result()
    next(q.get())

$ uname -a
Linux chaconne 5.10.4-arch2-1 #1 SMP PREEMPT Fri, 01 Jan 2021 05:29:53 +0000 x86_64 GNU/Linux
$ pip freeze
numpy==1.20.1
pyarrow==3.0.0

Issue Links

• links to: GitHub Pull Request #9480