The new pyarrow.fs FileSystem objects have a limited Python API (currently mimicking the C++ API).
In Python, fsspec defines a common API for a variety filesystem implementations.
We could try to implement a, fsspec-compatible class wrapping the pyarrow.fs native filesystems. Such as class would provide the methods expected according to fsspec, and implement those using the actual pyarrow.fs.FileSystem under the hood.
This might be mainly useful for two use cases:
- pyarrow.fs filesystems can be used in settings that expect an fsspec-compatible filesytem object
- it provides a way to have a "richer" API around our pyarrow.fs filesystems (which has been requested before, cfr
ARROW-7584), without expanding the core filesystem objects