Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-5874

[Python] pyarrow 0.14.0 macOS wheels depend on shared libs under /usr/local/opt

    XMLWordPrintableJSON

    Details

      Description

      Hello, and congrats on the recent release of Apache Arrow 0.14.0!

      This morning I installed pyarrow 0.14.0 on my macOS 10.14.5 system like so:

      python3.7 -m venv ~/virtualenv/pyarrow-0.14.0
      source ~/virtualenv/pyarrow-0.14.0/bin/activate
      pip install --upgrade pip setuptools
      pip install pyarrow  # installs pyarrow-0.14.0-cp37-cp37m-macosx_10_6_intel.whl
      
      pip freeze --all
      # numpy==1.16.4
      # pip==19.1.1
      # pyarrow==0.14.0
      # setuptools==41.0.1
      # six==1.12.0
      

      However I am unable to import pyarrow:

      python -c 'import pyarrow'
      # Traceback (most recent call last):
      #   File "<string>", line 1, in <module>
      #   File "/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/__init__.py", line 49, in <module>
      #     from pyarrow.lib import cpu_count, set_cpu_count
      # ImportError: dlopen(/Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/lib.cpython-37m-darwin.so, 2): Library not loaded: /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib
      #   Referenced from: /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
      #   Reason: image not found
      

      pyarrow is trying to load a shared library (OpenSSL in this case) from a path under /usr/local/opt that doesn't exist; perhaps that OpenSSL had been provided by Homebrew as part of your build process?  Unfortunately this makes the pyarrow 0.14.0 wheel completely unusable on my system or any system that doesn't have OpenSSL installed in that location.  This is a regression from pyarrow 0.13.0 as those wheels "just worked".

      Additional diagnostic output below.  I ran otool -L on each .dylib and .so file in /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow and included the output for those with dependencies under /usr/local/opt:

      otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib
      # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow.14.dylib:
      #         @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
      #         @rpath/libarrow_boost_system.dylib (compatibility version 0.0.0, current version 0.0.0)
      #         @rpath/libarrow_boost_filesystem.dylib (compatibility version 0.0.0, current version 0.0.0)
      #         @rpath/libarrow_boost_regex.dylib (compatibility version 0.0.0, current version 0.0.0)
      #         /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
      #         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
      
      
      otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib
      # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_flight.14.dylib:
      #         @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
      #         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
      
      otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib
      # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libarrow_python.14.dylib:
      #         @rpath/libarrow_python.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
      #         /System/Library/Frameworks/CoreFoundation.framework/Versions/A/CoreFoundation (compatibility version 150.0.0, current version 1349.64.0)
      #         @rpath/libarrow_flight.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
      
      otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libgandiva.14.dylib
      # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libgandiva.14.dylib:
      #         @rpath/libgandiva.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/lib/libz.1.dylib (compatibility version 1.0.0, current version 1.2.8)
      #         /usr/lib/libncurses.5.4.dylib (compatibility version 5.4.0, current version 5.4.0)
      #         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
      #         /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
      
      otool -L /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libplasma.14.dylib
      # /Users/manselmi/virtualenv/pyarrow-0.14.0/lib/python3.7/site-packages/pyarrow/libplasma.14.dylib:
      #         @rpath/libplasma.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         @rpath/libarrow.14.dylib (compatibility version 14.0.0, current version 14.0.0)
      #         /usr/local/opt/openssl/lib/libssl.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/local/opt/openssl/lib/libcrypto.1.0.0.dylib (compatibility version 1.0.0, current version 1.0.0)
      #         /usr/lib/libc++.1.dylib (compatibility version 1.0.0, current version 307.5.0)
      #         /usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1238.50.2)
      

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                kszucs Krisztian Szucs
                Reporter:
                manselmi Michael Anselmi
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 0.5h
                  0.5h