[ARROW-7102] [Python] Make filesystems compatible with fsspec

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: Python
• Labels: dataset-dask-integration filesystem
• External issue URL: https://github.com/apache/arrow/issues/23408
• Assignee: Unassigned
• Reporter: Tom Augspurger
• Created: 08/Nov/19 21:40
• Updated: 11/Jan/23 07:51
• Resolved: 20/Oct/21 14:06

Description

Update: regarding compatibility with fsspec, there are two directions of wrapping possible:

• Make a fsspec wrapper for pyarrow.fs (-> tracked in ARROW-8780, this can ensure pyarrow.fs filesystems can be used where fsspec filesytems are expected )
• Make a pyarrow.fs wrapper for fsspec (-> tracked in ARROW-8766 + ARROW-9089 this can ensure fsspec filesystems can be used where pyarrow.fs filesytems are expected )

---

fsspec defines a common API for a variety filesystem implementations. I'm proposing a FSSpecWrapper, similar to S3FSWrapper, that works with any fsspec implementation.

 

Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific to s3fs. https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320. This implementation could be removed entirely once an FSSPecWrapper is done, or kept as an alias if it's part of the public API.

 

This is realted to ARROW-3717, which requested a GCSFSWrapper for working with google cloud storage.

Issue Links

• relates to: ARROW-7584 [Python] Improve ergonomics of new FileSystem API

Sub-Tasks

• [Python] A FileSystem implementation based on Python callbacks: Resolved
• [Python] A fsspec-compatible wrapper for pyarrow.fs filesystems: Resolved
• [Python] A PyFileSystem handler for fsspec-based filesystems: Resolved
• [Python][Packaging] Enable S3 support in manylinux wheels: Resolved