[ARROW-8780] [Python] A fsspec-compatible wrapper for pyarrow.fs filesystems

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 6.0.0
• Component/s: Python
• Labels: documentation filesystem pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24926
• Assignee: Joris Van den Bossche
• Reporter: Joris Van den Bossche
• Created: 13/May/20 09:14
• Updated: 11/Jan/23 08:02
• Resolved: 18/Oct/21 11:28

Description

The new pyarrow.fs FileSystem objects have a limited Python API (currently mimicking the C++ API).

In Python, fsspec defines a common API for a variety filesystem implementations.

We could try to implement a, fsspec-compatible class wrapping the pyarrow.fs native filesystems. Such as class would provide the methods expected according to fsspec, and implement those using the actual pyarrow.fs.FileSystem under the hood.

This might be mainly useful for two use cases:

• pyarrow.fs filesystems can be used in settings that expect an fsspec-compatible filesytem object
• it provides a way to have a "richer" API around our pyarrow.fs filesystems (which has been requested before, cfr ARROW-7584), without expanding the core filesystem objects

Issue Links

• is related to: ARROW-7584 [Python] Improve ergonomics of new FileSystem API
• links to: GitHub Pull Request #11430