Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7102 [Python] Make filesystems compatible with fsspec
  3. ARROW-8780

[Python] A fsspec-compatible wrapper for pyarrow.fs filesystems

    XMLWordPrintableJSON

Details

    Description

      The new pyarrow.fs FileSystem objects have a limited Python API (currently mimicking the C++ API).

      In Python, fsspec defines a common API for a variety filesystem implementations.

      We could try to implement a, fsspec-compatible class wrapping the pyarrow.fs native filesystems. Such as class would provide the methods expected according to fsspec, and implement those using the actual pyarrow.fs.FileSystem under the hood.

      This might be mainly useful for two use cases:

      • pyarrow.fs filesystems can be used in settings that expect an fsspec-compatible filesytem object
      • it provides a way to have a "richer" API around our pyarrow.fs filesystems (which has been requested before, cfr ARROW-7584), without expanding the core filesystem objects

      Attachments

        Issue Links

          Activity

            People

              jorisvandenbossche Joris Van den Bossche
              jorisvandenbossche Joris Van den Bossche
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1h
                  1h