Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7102

[Python] Make filesystems compatible with fsspec

    XMLWordPrintableJSON

Details

    Description

      Update: regarding compatibility with fsspec, there are two directions of wrapping possible:

      • Make a fsspec wrapper for pyarrow.fs (-> tracked in ARROW-8780, this can ensure pyarrow.fs filesystems can be used where fsspec filesytems are expected )
      • Make a pyarrow.fs wrapper for fsspec (-> tracked in ARROW-8766 + ARROW-9089 this can ensure fsspec filesystems can be used where pyarrow.fs filesytems are expected )

      fsspec defines a common API for a variety filesystem implementations. I'm proposing a FSSpecWrapper, similar to S3FSWrapper, that works with any fsspec implementation.

       

      Right now, pyarrow has a pyarrow.filesystems.S3FSWrapper, which is specific to s3fs. https://github.com/apache/arrow/blob/21ad7ac1162eab188a1e15923fb1de5b795337ec/python/pyarrow/filesystem.py#L320. This implementation could be removed entirely once an FSSPecWrapper is done, or kept as an alias if it's part of the public API.

       

      This is realted to ARROW-3717, which requested a GCSFSWrapper for working with google cloud storage.

      Attachments

        Issue Links

          There are no Sub-Tasks for this issue.

          Activity

            People

              Unassigned Unassigned
              TomAugspurger Tom Augspurger
              Votes:
              0 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 8h 50m
                  8h 50m