[ARROW-10370] [Python] Spurious s3fs-related test failures

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/26356
• Assignee: Joris Van den Bossche
• Reporter: Antoine Pitrou
• Created: 22/Oct/20 13:58
• Updated: 11/Jan/23 08:12
• Resolved: 25/Jan/21 15:04

Description

I frequently get this error when running the Python test suite:

_____________________________________________________________ test_write_to_dataset_pathlib_nonlocal[False] _____________________________________________________________
Traceback (most recent call last):
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 984, in _initiate_upload
    Bucket=self.bucket, Key=self.key, ACL=self.acl)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 971, in _call_s3
    **kwargs)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 189, in _call_s3
    return method(**additional_kwargs)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
    return self._make_api_call(operation_name, kwargs)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
    raise error_class(parsed_response, operation_name)
botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the CreateMultipartUpload operation: The specified bucket does not exist

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/antoine/arrow/dev/python/pyarrow/tests/test_parquet.py", line 2978, in test_write_to_dataset_pathlib_nonlocal
    tempdir / "test1", use_legacy_dataset, filesystem=fs)
  File "/home/antoine/arrow/dev/python/pyarrow/tests/test_parquet.py", line 2853, in _test_write_to_dataset_with_partitions
    pq.write_metadata(output_table.schema, f)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1457, in __exit__
    self.close()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
    self.flush(force=True)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
    self._initiate_upload()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
    raise translate_boto_error(e)
FileNotFoundError: The specified bucket does not exist
------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------
Exception ignored in: Exception ignored in: Exception ignored in: <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
Traceback (most recent call last):
<function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
Exception ignored in:   File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
<function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
Traceback (most recent call last):
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
<function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
Exception ignored in: Traceback (most recent call last):
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
Traceback (most recent call last):
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
<function AbstractBufferedFile.__del__ at 0x7f1b119097a0>    Exception ignored in: <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>    self.close()
self.close()
Traceback (most recent call last):
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close

  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
        self.flush(force=True)    
self.flush(force=True)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush

self.close()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
            self._initiate_upload()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
Traceback (most recent call last):
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
self.close()self.flush(force=True)    self._initiate_upload()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload


  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
      File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
            raise translate_boto_error(e)self.close()self.close()    
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close

  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
raise translate_boto_error(e)
self._initiate_upload()
FileNotFoundError: 
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
The specified bucket does not existFileNotFoundError: 
The specified bucket does not exist
        self.flush(force=True)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
self.flush(force=True)
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
    self.flush(force=True)
      File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
    self._initiate_upload()raise translate_boto_error(e)
FileNotFoundError: The specified bucket does not exist

  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
        self._initiate_upload()
self._initiate_upload()
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
  File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
    raise translate_boto_error(e)
FileNotFoundError: The specified bucket does not exist
        raise translate_boto_error(e)raise translate_boto_error(e)

FileNotFoundErrorFileNotFoundError: : The specified bucket does not existThe specified bucket does not exist

Issue Links

• links to: GitHub Pull Request #9284