Update: regarding compatibility with fsspec, there are two directions of wrapping possible:
- Make a fsspec wrapper for pyarrow.fs (-> tracked in
ARROW-8780, this can ensure pyarrow.fs filesystems can be used where fsspec filesytems are expected )
- Make a pyarrow.fs wrapper for fsspec (-> tracked in
ARROW-8766+ ARROW-9089this can ensure fsspec filesystems can be used where pyarrow.fs filesytems are expected )
fsspec defines a common API for a variety filesystem implementations. I'm proposing a FSSpecWrapper, similar to S3FSWrapper, that works with any fsspec implementation.
Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific to s3fs. https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320. This implementation could be removed entirely once an FSSPecWrapper is done, or kept as an alias if it's part of the public API.
This is realted to
ARROW-3717, which requested a GCSFSWrapper for working with google cloud storage.