Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-5072

[Python] write_table fails silently on S3 errors

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Minor
    • Resolution: Won't Fix
    • Affects Version/s: 0.12.1
    • Fix Version/s: 0.15.0
    • Component/s: Python
    • Environment:
      Python 3.6.8

      Description

      pyarrow==0.12.1

      pyarrow.parquet.write_table called with where=S3File(...) fails silently when encountering errors while writing to S3 (in the example below, boto3 is raising a NoSuchBucket exception). However, instead of using S3File(), calling write_table with where=<filepath> and with filesystem=S3FileSystem() does not fail silently and raises, as is expected.

      Code/Repro

       

      import pandas as pd
      import pyarrow as pa
      import pyarrow.parquet as pq
      from s3fs import S3File, S3FileSystem
      
      
      df = pd.DataFrame({'col0': []})
      s3_filepath = 's3://some-bogus-bucket/df.parquet'
      
      print('>> test 1')
      try:
      # use S3File --> fails silently
      pq.write_table(pa.Table.from_pandas(df.copy()),
      S3File(S3FileSystem(), s3_filepath, mode='wb'))
      except Exception:
      print('>>>> Exception raised!')
      else:
      print('>>>> Exception **NOT** raised!')
      
      
      print('>> test 2')
      try:
      # use filepath and S3FileSystem --> raises Exception, as expected
      pq.write_table(pa.Table.from_pandas(df.copy()),
      s3_filepath,
      filesystem=S3FileSystem())
      except Exception:
      print('>>>> Exception raised!')
      else:
      print('>>>> Exception **NOT** raised!')

       

       

      Output

      >> test 1
      Exception ignored in: <bound method S3File.__del__ of <S3File some-bogus-bucket/df.parquet>>
      Traceback (most recent call last):
      File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in __del__
      self.close()
      File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in close
      raise_from(IOError('Write failed: %s' % self.path), e)
      File "<string>", line 3, in raise_from
      OSError: Write failed: some-bogus-bucket/df.parquet
      >>>> Exception **NOT** raised!
      
      >> test 2
      >>>> Exception raised!
      Exception ignored in: <bound method S3File.__del__ of <S3File some-bogus-bucket/df.parquet>>
      Traceback (most recent call last):
      File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in __del__
      self.close()
      File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in close
      raise_from(IOError('Write failed: %s' % self.path), e)
      File "<string>", line 3, in raise_from
      OSError: Write failed: some-bogus-bucket/df.parquet
      

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              pgeez Paul George
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: