[ARROW-11538] [Python] Segfault reading Parquet dataset with Timestamp filter

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 4.0.0
• Component/s: Python
• Environment: platform: Linux 64bit ; conda env: ; conda create -n pya python=3.8 pyarrow=3.0.0 pandas=1.2.1 pytest -c conda-forge
• External issue URL: https://github.com/apache/arrow/issues/27412
• Assignee: Joris Van den Bossche
• Reporter: Josh
• Created: 06/Feb/21 15:01
• Updated: 11/Jan/23 08:20
• Resolved: 11/Feb/21 14:55

Description

The first two tests pass but the third gives: Fatal Python error: Segmentation fault

All three pass in with pyarrow=2.0.0

import pandas
import pyarrow as pa
import pyarrow.dataset as ds
import pyarrow.parquet as pq
import pytest


@pytest.fixture
def data_path(tmp_path):
    path = tmp_path / "data.parquet"
    df = pandas.DataFrame(
        [
            ["A", pandas.Timestamp("2020-11-04")],
        ],
        columns=["name", "date"],
    )
    table = pa.Table.from_pandas(df)
    pq.write_table(table, path, version="2.0")
    return df, path


@pytest.mark.parametrize(
    "filter",
    [
        None,
        ds.field("date") == "2020-11-04",
        ds.field("date") == pandas.Timestamp("2020-11-04"),
    ],
)
def test_dataset_filter(filter, data_path):
    data, path = data_path

    dataset = ds.dataset(path, format="parquet")
    assert data.equals(dataset.to_table(filter=filter).to_pandas())

Issue Links

• is duplicated by: ARROW-11480 [Python] Segmentation fault reading parquet with date filter with INT96 column