[ARROW-17791] [Python][CI] Some nightly jobs are failing due to ACCESS_DENIED to S3 bucket

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 10.0.0
• Component/s: Continuous Integration, Python
• Labels: Nightly
• External issue URL: https://github.com/apache/arrow/issues/33017
• Assignee: Jacob Wujciak
• Reporter: Raúl Cumplido
• Created: 21/Sep/22 08:33
• Updated: 11/Jan/23 11:55
• Resolved: 23/Sep/22 13:54

Description

The following nitghly failures:

• test-conda-python-3.10
• test-conda-python-3.7
• test-conda-python-3.7-pandas-0.24
• test-conda-python-3.7-pandas-latest
• test-conda-python-3.8
• test-conda-python-3.8-pandas-latest
• test-conda-python-3.8-pandas-nightly
• test-conda-python-3.9
• test-conda-python-3.9-pandas-master
• test-debian-11-python-3

Failed Python test_s3_real_aws_region_selection with ACCESS_DENIED:

 =================================== FAILURES ===================================
______________________ test_s3_real_aws_region_selection _______________________    @pytest.mark.s3
    def test_s3_real_aws_region_selection():
        # Taken from a registry of open S3-hosted datasets
        # at https://github.com/awslabs/open-data-registry
        fs, path = FileSystem.from_uri('s3://mf-nwp-models/README.txt')
        assert fs.region == 'eu-west-1'
>       with fs.open_input_stream(path) as f:opt/conda/envs/arrow/lib/python3.10/site-packages/pyarrow/tests/test_fs.py:1660: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
pyarrow/_fs.pyx:805: in pyarrow._fs.FileSystem.open_input_stream
    ???
pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
    ???
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >   ???
E   OSError: When reading information for key 'README.txt' in bucket 'mf-nwp-models': AWS Error ACCESS_DENIED during HeadObject operation: No response body.pyarrow/error.pxi:115: OSError