[ARROW-10921] `TypeError: 'coroutine' object is not iterable` when reading parquet partitions via s3fs >= 0.5 with pyarrow

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Python
• External issue URL: https://github.com/apache/arrow/issues/26849
• Assignee: Unassigned
• Reporter: Ivan Necas
• Created: 15/Dec/20 16:10
• Updated: 11/Jan/23 08:16
• Resolved: 15/Dec/20 16:29

Description

Trying to read partitioned parquet files using updated s3fs >= 0.5 (using via }}{{dask), and got this error:

 

410 files = set()
 411 
--> 412 for key in list(self.fs._ls(path, refresh=refresh)):
 413 path = key['Key']
 414 if key['StorageClass'] == 'DIRECTORY':

TypeError: 'coroutine' object is not iterable

coming from https://github.com/apache/arrow/blob/9baa123ea38ee9cc1d3a90cfc9347239cd28064c/python/pyarrow/filesystem.py#L415 

 

Seems related to switching s3fs to asyncio in https://github.com/dask/s3fs/pull/336.

Issue Links

• is duplicated by: ARROW-10433 [Python] pyarrow doesn't work with s3fs>=0.5
• relates to: ARROW-10433 [Python] pyarrow doesn't work with s3fs>=0.5