Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-5158

[Packaging][Wheel] Symlink libraries in wheels

Agile BoardAttach filesAttach ScreenshotVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 1.0.0
    • Packaging, Python

    Description

      Libraries are copied instead of symlinking in linux and osx wheels, which result quiet big binaries:

       
      This is what the wheel contains before running auditwheel:
       

      -rwxr-xr-x  1 root root 128K Apr  3 09:02 libarrow_boost_filesystem.so
      -rwxr-xr-x  1 root root 128K Apr  3 09:02 libarrow_boost_filesystem.so.1.66.0
      -rwxr-xr-x  1 root root 1.2M Apr  3 09:02 libarrow_boost_regex.so
      -rwxr-xr-x  1 root root 1.2M Apr  3 09:02 libarrow_boost_regex.so.1.66.0
      -rwxr-xr-x  1 root root  30K Apr  3 09:02 libarrow_boost_system.so
      -rwxr-xr-x  1 root root  30K Apr  3 09:02 libarrow_boost_system.so.1.66.0
      -rwxr-xr-x  1 root root 1.4M Apr  3 09:02 libarrow_python.so
      -rwxr-xr-x  1 root root 1.4M Apr  3 09:02 libarrow_python.so.14
      -rwxr-xr-x  1 root root  12M Apr  3 09:02 libarrow.so
      -rwxr-xr-x  1 root root  12M Apr  3 09:02 libarrow.so.14
      -rw-r--r--  1 root root 6.1M Apr  3 09:02 lib.cpp
      -rwxr-xr-x  1 root root 2.4M Apr  3 09:02 [lib.cpython-36m-x86_64-linux-gnu.so|http://lib.cpython-36m-x86_64-linux-gnu.so/]
      -rwxr-xr-x  1 root root  55M Apr  3 09:02 libgandiva.so
      -rwxr-xr-x  1 root root  55M Apr  3 09:02 libgandiva.so.14
      -rwxr-xr-x  1 root root 2.9M Apr  3 09:02 libparquet.so
      -rwxr-xr-x  1 root root 2.9M Apr  3 09:02 libparquet.so.14
      -rwxr-xr-x  1 root root 309K Apr  3 09:02 libplasma.so
      -rwxr-xr-x  1 root root 309K Apr  3 09:02 libplasma.so.14
       

      After running auditwheel, the repaired wheel contains:
       

      -rwxr-xr-x  1 root root 128K Apr  3 09:02 libarrow_boost_filesystem.so
      -rwxr-xr-x  1 root root 128K Apr  3 09:02 libarrow_boost_filesystem.so.1.66.0
      -rwxr-xr-x  1 root root 1.2M Apr  3 09:02 libarrow_boost_regex.so
      -rwxr-xr-x  1 root root 1.2M Apr  3 09:02 libarrow_boost_regex.so.1.66.0
      -rwxr-xr-x  1 root root  30K Apr  3 09:02 libarrow_boost_system.so
      -rwxr-xr-x  1 root root  30K Apr  3 09:02 libarrow_boost_system.so.1.66.0
      -rwxr-xr-x  1 root root 1.6M Apr  3 09:55 libarrow_python.so
      -rwxr-xr-x  1 root root 1.4M Apr  3 09:02 libarrow_python.so.14
      -rwxr-xr-x  1 root root  12M Apr  3 09:55 libarrow.so
      -rwxr-xr-x  1 root root  12M Apr  3 09:02 libarrow.so.14
      -rw-r--r--  1 root root 6.1M Apr  3 09:02 lib.cpp
      -rwxr-xr-x  1 root root 2.5M Apr  3 09:55 [lib.cpython-36m-x86_64-linux-gnu.so|http://lib.cpython-36m-x86_64-linux-gnu.so/]
      -rwxr-xr-x  1 root root  59M Apr  3 09:55 libgandiva.so
      -rwxr-xr-x  1 root root  55M Apr  3 09:02 libgandiva.so.14
      -rwxr-xr-x  1 root root 3.5M Apr  3 09:55 libparquet.so
      -rwxr-xr-x  1 root root 2.9M Apr  3 09:02 libparquet.so.14
      -rwxr-xr-x  1 root root 345K Apr  3 09:55 libplasma.so
      -rwxr-xr-x  1 root root 309K Apr  3 09:02 libplasma.so.14
      

       
      Here is the output of auditwheel https://travis-ci.org/kszucs/crossbow/builds/514605723#L3340

      They should be symlinks, we have special code for this: https://github.com/apache/arrow/blob/4495305092411e8551c60341e273c8aa3c14b282/python/setup.py#L489-L499 This is probably not going into the wheel as wheels are zip-files and they don't support symlinks by default. So we probably need to pass the `--symlinks` parameter to the wheel code.

      Attachments

        Issue Links

        Activity

          This comment will be Viewable by All Users Viewable by All Users
          Cancel

          People

            wesm Wes McKinney Assign to me
            kszucs Krisztian Szucs
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Slack

                Issue deployment