Details
-
Bug
-
Status: Closed
-
Minor
-
Resolution: Won't Fix
-
0.12.1
-
Python 3.6.8
Description
pyarrow==0.12.1
pyarrow.parquet.write_table called with where=S3File(...) fails silently when encountering errors while writing to S3 (in the example below, boto3 is raising a NoSuchBucket exception). However, instead of using S3File(), calling write_table with where=<filepath> and with filesystem=S3FileSystem() does not fail silently and raises, as is expected.
Code/Repro
import pandas as pd import pyarrow as pa import pyarrow.parquet as pq from s3fs import S3File, S3FileSystem df = pd.DataFrame({'col0': []}) s3_filepath = 's3://some-bogus-bucket/df.parquet' print('>> test 1') try: # use S3File --> fails silently pq.write_table(pa.Table.from_pandas(df.copy()), S3File(S3FileSystem(), s3_filepath, mode='wb')) except Exception: print('>>>> Exception raised!') else: print('>>>> Exception **NOT** raised!') print('>> test 2') try: # use filepath and S3FileSystem --> raises Exception, as expected pq.write_table(pa.Table.from_pandas(df.copy()), s3_filepath, filesystem=S3FileSystem()) except Exception: print('>>>> Exception raised!') else: print('>>>> Exception **NOT** raised!')
Output
>> test 1 Exception ignored in: <bound method S3File.__del__ of <S3File some-bogus-bucket/df.parquet>> Traceback (most recent call last): File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in __del__ self.close() File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in close raise_from(IOError('Write failed: %s' % self.path), e) File "<string>", line 3, in raise_from OSError: Write failed: some-bogus-bucket/df.parquet >>>> Exception **NOT** raised! >> test 2 >>>> Exception raised! Exception ignored in: <bound method S3File.__del__ of <S3File some-bogus-bucket/df.parquet>> Traceback (most recent call last): File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1476, in __del__ self.close() File "<redacted>/lib/python3.6/site-packages/s3fs/core.py", line 1454, in close raise_from(IOError('Write failed: %s' % self.path), e) File "<string>", line 3, in raise_from OSError: Write failed: some-bogus-bucket/df.parquet