[ARROW-10433] [Python] pyarrow doesn't work with s3fs>=0.5

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 3.0.0
• Component/s: Python
• Labels: pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/18347
• Assignee: Marius van Niekerk
• Reporter: Marius van Niekerk
• Created: 30/Oct/20 18:06
• Updated: 11/Jan/23 08:13
• Resolved: 10/Nov/20 15:58

Description

s3fs has moved to using asyncio underneath. 

Unfortunately pyarrow relies on internal private methods

as seen here 
https://github.com/apache/arrow/blob/478286658055bb91737394c2065b92a7e92fb0c1/python/pyarrow/filesystem.py#L412

this _ls has been changed to be an asynchronous coroutine in more modern versions of s3fs

 

Issue Links

• duplicates: ARROW-10921 `TypeError: 'coroutine' object is not iterable` when reading parquet partitions via s3fs >= 0.5 with pyarrow
• is related to: ARROW-10921 `TypeError: 'coroutine' object is not iterable` when reading parquet partitions via s3fs >= 0.5 with pyarrow
• links to: GitHub Pull Request #8557
• links to: https://github.com/apache/arrow/pull/8557