Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-18392

[CI][Python] Some nightly python tests fail due to ACCESS DENIED to S3 bucket

Details

    Description

      Several nightly tests fail with:

       =================================== FAILURES ===================================
      ____________________________ test_s3fs_wrong_region ____________________________    @pytest.mark.s3
          def test_s3fs_wrong_region():
              from pyarrow.fs import S3FileSystem
          
              # wrong region for bucket
              fs = S3FileSystem(region='eu-north-1')
          
              msg = ("When getting information for bucket 'voltrondata-labs-datasets': "
                     r"AWS Error UNKNOWN \(HTTP status 301\) during HeadBucket "
                     "operation: No response body. Looks like the configured region is "
                     "'eu-north-1' while the bucket is located in 'us-east-2'."
                     "|NETWORK_CONNECTION")
              with pytest.raises(OSError, match=msg) as exc:
                  fs.get_file_info("voltrondata-labs-datasets")
          
              # Sometimes fails on unrelated network error, so next call would also fail.
              if 'NETWORK_CONNECTION' in str(exc.value):
                  return
          
              fs = S3FileSystem(region='us-east-2')
      >       fs.get_file_info("voltrondata-labs-datasets")opt/conda/envs/arrow/lib/python3.7/site-packages/pyarrow/tests/test_fs.py:1339: 
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
      pyarrow/_fs.pyx:571: in pyarrow._fs.FileSystem.get_file_info
          ???
      pyarrow/error.pxi:144: in pyarrow.lib.pyarrow_internal_check_status
          ???
      _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ >   ???
      E   OSError: When getting information for bucket 'voltrondata-labs-datasets': AWS Error ACCESS_DENIED during HeadBucket operation: No response body. 

      I can't seem to be able to reproduce locally but is pretty consistent:

      Attachments

        Issue Links

          Activity

            People

              milesgranger Miles Granger
              raulcd Raúl Cumplido
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 2h 10m
                  2h 10m

                  Slack

                    Issue deployment