[ARROW-17614] [CI][Python] test test_write_dataset_max_rows_per_file is producing several nightly build failures

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 10.0.0
• Component/s: Continuous Integration, Python
• Labels: Nightly pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/32858
• Assignee: Weston Pace
• Reporter: Raúl Cumplido
• Created: 05/Sep/22 07:38
• Updated: 11/Jan/23 11:51
• Resolved: 26/Sep/22 16:24

Description

The following failure has been seen on multiple nightly builds:

_____________________ test_write_dataset_max_rows_per_file _____________________tempdir = PosixPath('/tmp/pytest-of-root/pytest-0/test_write_dataset_max_rows_pe0')    @pytest.mark.parquet
    def test_write_dataset_max_rows_per_file(tempdir):
        directory = tempdir / 'ds'
        max_rows_per_file = 10
        max_rows_per_group = 10
        num_of_columns = 2
        num_of_records = 35
    
        record_batch = _generate_data_and_columns(num_of_columns,
                                                  num_of_records)
    
        ds.write_dataset(record_batch, directory, format="parquet",
                         max_rows_per_file=max_rows_per_file,
>                        max_rows_per_group=max_rows_per_group)usr/local/lib/python3.7/site-packages/pyarrow/tests/test_dataset.py:3921: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
usr/local/lib/python3.7/site-packages/pyarrow/dataset.py:992: in write_dataset
    min_rows_per_group, max_rows_per_group, create_dir
pyarrow/_dataset.pyx:2811: in pyarrow._dataset._filesystemdataset_write
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >   ???
E   FileNotFoundError: [Errno 2] Failed to open local file '/tmp/pytest-of-root/pytest-0/test_write_dataset_max_rows_pe0/ds/part-1.parquet'. Detail: [errno 2] No such file or directory 

Example of failed builds:

verify-rc-source-python-macos-conda-amd64

wheel-manylinux2014-cp37-amd64

It seems flaky as there were some nightly jobs executed on a previous day without new commits that were successful.

Issue Links

• causes: ARROW-17853 [Python][CI] Timeout in test_dataset.py::test_write_dataset_s3_put_only
• supercedes: ARROW-17633 [Python][CI] test_write_dataset_max_rows_per_file is flaky
• links to: GitHub Pull Request #14199