Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-8518

[Python] Create tools to enable optional components (like Gandiva, Flight) to be built and deployed as separate Python packages

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    Description

      Our current monolithic approach to Python packaging isn't likely to be sustainable long-term.

      At a high level, I would propose a structure like this:

      pip install pyarrow  # core package containing libarrow, libarrow_python, and any other common bundled C++ library dependencies
      
      pip install pyarrow-flight  # installs pyarrow, pyarrow_flight
      
      pip install pyarrow-gandiva # installs pyarrow, pyarrow_gandiva
      

      We can maintain the semantic appearance of a single pyarrow package by having thin API modules that would look like

      CONTENTS OF pyarrow/flight.py
      
      from pyarrow_flight import *
      

      Obviously, this is more difficult to build and package:

      • CMake and setup.py files must be refactored a bit so that we can reuse code between the parent and child packages
      • Separate conda and wheel packages must be produced. With conda this seems more straightforward but since the child wheels depend on the parent core wheel, the build process seems more complicated

      In any case, I don't think these challenges are insurmountable. This will have several benefits:

      • Smaller installation footprint for simple use cases (though note we are STILL duplicating shared libraries in the wheels, which is quite bad)
      • Less developer anxiety about expanding the scope of what Python code is shipped from apache/arrow. If in 5 years we are shipping 5 different Python wheels with each Apache Arrow release, that sounds completely fine to me.

      Attachments

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            Unassigned Unassigned
            wesm Wes McKinney

            Dates

              Created:
              Updated:

              Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 1h
                1h

                Slack

                  Issue deployment