Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-10370

[Python] Spurious s3fs-related test failures

    XMLWordPrintableJSON

Details

    Description

      I frequently get this error when running the Python test suite:

      _____________________________________________________________ test_write_to_dataset_pathlib_nonlocal[False] _____________________________________________________________
      Traceback (most recent call last):
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 984, in _initiate_upload
          Bucket=self.bucket, Key=self.key, ACL=self.acl)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 971, in _call_s3
          **kwargs)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 189, in _call_s3
          return method(**additional_kwargs)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
          return self._make_api_call(operation_name, kwargs)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/botocore/client.py", line 661, in _make_api_call
          raise error_class(parsed_response, operation_name)
      botocore.errorfactory.NoSuchBucket: An error occurred (NoSuchBucket) when calling the CreateMultipartUpload operation: The specified bucket does not exist
      
      During handling of the above exception, another exception occurred:
      
      Traceback (most recent call last):
        File "/home/antoine/arrow/dev/python/pyarrow/tests/test_parquet.py", line 2978, in test_write_to_dataset_pathlib_nonlocal
          tempdir / "test1", use_legacy_dataset, filesystem=fs)
        File "/home/antoine/arrow/dev/python/pyarrow/tests/test_parquet.py", line 2853, in _test_write_to_dataset_with_partitions
          pq.write_metadata(output_table.schema, f)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1457, in __exit__
          self.close()
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
          self.flush(force=True)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
          self._initiate_upload()
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
          raise translate_boto_error(e)
      FileNotFoundError: The specified bucket does not exist
      ------------------------------------------------------------------------- Captured stderr call --------------------------------------------------------------------------
      Exception ignored in: Exception ignored in: Exception ignored in: <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
      Traceback (most recent call last):
      <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
      Exception ignored in:   File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
      <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
      Traceback (most recent call last):
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
      <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>
      Exception ignored in: Traceback (most recent call last):
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
      Traceback (most recent call last):
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
      <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>    Exception ignored in: <function AbstractBufferedFile.__del__ at 0x7f1b119097a0>    self.close()
      self.close()
      Traceback (most recent call last):
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
      
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
              self.flush(force=True)    
      self.flush(force=True)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
      
      self.close()
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
                  self._initiate_upload()
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
      Traceback (most recent call last):
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1446, in __del__
      self.close()self.flush(force=True)    self._initiate_upload()
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
      
      
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
            File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
                  raise translate_boto_error(e)self.close()self.close()    
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
      
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1425, in close
      raise translate_boto_error(e)
      self._initiate_upload()
      FileNotFoundError: 
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
      The specified bucket does not existFileNotFoundError: 
      The specified bucket does not exist
              self.flush(force=True)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
      self.flush(force=True)
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
          self.flush(force=True)
            File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/fsspec/spec.py", line 1297, in flush
          self._initiate_upload()raise translate_boto_error(e)
      FileNotFoundError: The specified bucket does not exist
      
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
              self._initiate_upload()
      self._initiate_upload()
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
        File "/home/antoine/miniconda3/envs/pyarrow/lib/python3.7/site-packages/s3fs/core.py", line 986, in _initiate_upload
          raise translate_boto_error(e)
      FileNotFoundError: The specified bucket does not exist
              raise translate_boto_error(e)raise translate_boto_error(e)
      
      FileNotFoundErrorFileNotFoundError: : The specified bucket does not existThe specified bucket does not exist
      
      

      Attachments

        Issue Links

          Activity

            People

              jorisvandenbossche Joris Van den Bossche
              apitrou Antoine Pitrou
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3h
                  3h