[ARROW-8766] [Python] A FileSystem implementation based on Python callbacks

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Python
• Labels: dataset-dask-integration filesystem pull-request-available
• External issue URL: https://github.com/apache/arrow/issues/24914
• Assignee: Antoine Pitrou
• Reporter: Joris Van den Bossche
• Created: 11/May/20 15:44
• Updated: 11/Jan/23 08:02
• Resolved: 09/Jun/20 17:01

Description

The new pyarrow.fs filesystems are now actual C++ objects, and no longer "just" a python interface. So they can't easily be expanded from the Python side, and the existing integration with fsspec filesystems is therefore also not working anymore.

One possible solution is to have a C++ filesystem that calls back into a python object for each of its methods (possibly similar to how you can implement a flight server in Python, I suppose).

Such a FileSystem implementation would allow to make a pyarrow.fs wrapper for fsspec filesystems, and thus allow such filesystems to be used in pyarrow where new filesystems are expected.

Issue Links

• links to: GitHub Pull Request #7349