[ARROW-18088] [Python][CI] Build with pandas master/nightly failure related to timedelta64 resolution

• Type: Test
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 10.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/33286
• Assignee: Joris Van den Bossche
• Reporter: Joris Van den Bossche
• Created: 18/Oct/22 16:05
• Updated: 11/Jan/23 11:58
• Resolved: 21/Oct/22 12:56

Description

The nightly python builds using the pandas development version are failing: https://github.com/ursacomputing/crossbow/actions/runs/3269767207/jobs/5377649455

Example failure:

 ____________________ test_parquet_2_0_roundtrip[None-True] _____________________

tempdir = PosixPath('/tmp/pytest-of-root/pytest-0/test_parquet_2_0_roundtrip_Non0')
chunk_size = None, use_legacy_dataset = True

    @pytest.mark.pandas
    @parametrize_legacy_dataset
    @pytest.mark.parametrize('chunk_size', [None, 1000])
    def test_parquet_2_0_roundtrip(tempdir, chunk_size, use_legacy_dataset):
        df = alltypes_sample(size=10000, categorical=True)
    
        filename = tempdir / 'pandas_roundtrip.parquet'
        arrow_table = pa.Table.from_pandas(df)
        assert arrow_table.schema.pandas_metadata is not None
    
        _write_table(arrow_table, filename, version='2.6',
                     coerce_timestamps='ms', chunk_size=chunk_size)
        table_read = pq.read_pandas(
            filename, use_legacy_dataset=use_legacy_dataset)
        assert table_read.schema.pandas_metadata is not None
    
        read_metadata = table_read.schema.metadata
        assert arrow_table.schema.metadata == read_metadata
    
        df_read = table_read.to_pandas()
>       tm.assert_frame_equal(df, df_read)
E       AssertionError: Attributes of DataFrame.iloc[:, 12] (column name="timedelta") are different
E       
E       Attribute "dtype" are different
E       [left]:  timedelta64[s]
E       [right]: timedelta64[ns]

opt/conda/envs/arrow/lib/python3.9/site-packages/pyarrow/tests/parquet/test_data_types.py:76: AssertionError

Issue Links

• relates to: ARROW-18124 [Python] Support converting to non-nano datetime64 for pandas >= 2.0
• links to: GitHub Pull Request #14460