Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-7102 [Python] Make filesystems compatible with fsspec
  3. ARROW-8766

[Python] A FileSystem implementation based on Python callbacks

    XMLWordPrintableJSON

Details

    Description

      The new pyarrow.fs filesystems are now actual C++ objects, and no longer "just" a python interface. So they can't easily be expanded from the Python side, and the existing integration with fsspec filesystems is therefore also not working anymore.

      One possible solution is to have a C++ filesystem that calls back into a python object for each of its methods (possibly similar to how you can implement a flight server in Python, I suppose).

      Such a FileSystem implementation would allow to make a pyarrow.fs wrapper for fsspec filesystems, and thus allow such filesystems to be used in pyarrow where new filesystems are expected.

      Attachments

        Issue Links

          Activity

            People

              apitrou Antoine Pitrou
              jorisvandenbossche Joris Van den Bossche
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 1.5h
                  1.5h