[ARROW-5158] [Packaging][Wheel] Symlink libraries in wheels

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Packaging, Python
• Labels: wheel
• External issue URL: https://github.com/apache/arrow/issues/21637
• Assignee: Wes McKinney
• Reporter: Krisztian Szucs
• Created: 10/Apr/19 12:11
• Updated: 11/Jan/23 07:38
• Resolved: 15/Jun/20 16:23

Description

Libraries are copied instead of symlinking in linux and osx wheels, which result quiet big binaries:

 
This is what the wheel contains before running auditwheel:
 

-rwxr-xr-x  1 root root 128K Apr  3 09:02 libarrow_boost_filesystem.so
-rwxr-xr-x  1 root root 128K Apr  3 09:02 libarrow_boost_filesystem.so.1.66.0
-rwxr-xr-x  1 root root 1.2M Apr  3 09:02 libarrow_boost_regex.so
-rwxr-xr-x  1 root root 1.2M Apr  3 09:02 libarrow_boost_regex.so.1.66.0
-rwxr-xr-x  1 root root  30K Apr  3 09:02 libarrow_boost_system.so
-rwxr-xr-x  1 root root  30K Apr  3 09:02 libarrow_boost_system.so.1.66.0
-rwxr-xr-x  1 root root 1.4M Apr  3 09:02 libarrow_python.so
-rwxr-xr-x  1 root root 1.4M Apr  3 09:02 libarrow_python.so.14
-rwxr-xr-x  1 root root  12M Apr  3 09:02 libarrow.so
-rwxr-xr-x  1 root root  12M Apr  3 09:02 libarrow.so.14
-rw-r--r--  1 root root 6.1M Apr  3 09:02 lib.cpp
-rwxr-xr-x  1 root root 2.4M Apr  3 09:02 [lib.cpython-36m-x86_64-linux-gnu.so|http://lib.cpython-36m-x86_64-linux-gnu.so/]
-rwxr-xr-x  1 root root  55M Apr  3 09:02 libgandiva.so
-rwxr-xr-x  1 root root  55M Apr  3 09:02 libgandiva.so.14
-rwxr-xr-x  1 root root 2.9M Apr  3 09:02 libparquet.so
-rwxr-xr-x  1 root root 2.9M Apr  3 09:02 libparquet.so.14
-rwxr-xr-x  1 root root 309K Apr  3 09:02 libplasma.so
-rwxr-xr-x  1 root root 309K Apr  3 09:02 libplasma.so.14
 

After running auditwheel, the repaired wheel contains:
 

-rwxr-xr-x  1 root root 128K Apr  3 09:02 libarrow_boost_filesystem.so
-rwxr-xr-x  1 root root 128K Apr  3 09:02 libarrow_boost_filesystem.so.1.66.0
-rwxr-xr-x  1 root root 1.2M Apr  3 09:02 libarrow_boost_regex.so
-rwxr-xr-x  1 root root 1.2M Apr  3 09:02 libarrow_boost_regex.so.1.66.0
-rwxr-xr-x  1 root root  30K Apr  3 09:02 libarrow_boost_system.so
-rwxr-xr-x  1 root root  30K Apr  3 09:02 libarrow_boost_system.so.1.66.0
-rwxr-xr-x  1 root root 1.6M Apr  3 09:55 libarrow_python.so
-rwxr-xr-x  1 root root 1.4M Apr  3 09:02 libarrow_python.so.14
-rwxr-xr-x  1 root root  12M Apr  3 09:55 libarrow.so
-rwxr-xr-x  1 root root  12M Apr  3 09:02 libarrow.so.14
-rw-r--r--  1 root root 6.1M Apr  3 09:02 lib.cpp
-rwxr-xr-x  1 root root 2.5M Apr  3 09:55 [lib.cpython-36m-x86_64-linux-gnu.so|http://lib.cpython-36m-x86_64-linux-gnu.so/]
-rwxr-xr-x  1 root root  59M Apr  3 09:55 libgandiva.so
-rwxr-xr-x  1 root root  55M Apr  3 09:02 libgandiva.so.14
-rwxr-xr-x  1 root root 3.5M Apr  3 09:55 libparquet.so
-rwxr-xr-x  1 root root 2.9M Apr  3 09:02 libparquet.so.14
-rwxr-xr-x  1 root root 345K Apr  3 09:55 libplasma.so
-rwxr-xr-x  1 root root 309K Apr  3 09:02 libplasma.so.14

 
Here is the output of auditwheel https://travis-ci.org/kszucs/crossbow/builds/514605723#L3340

They should be symlinks, we have special code for this: https://github.com/apache/arrow/blob/4495305092411e8551c60341e273c8aa3c14b282/python/setup.py#L489-L499 This is probably not going into the wheel as wheels are zip-files and they don't support symlinks by default. So we probably need to pass the `--symlinks` parameter to the wheel code.

Issue Links

• relates to: ARROW-5980 [Python] Missing libarrow.so and libarrow_python.so in wheel file