Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-2269

[Python] Cannot build bdist_wheel for Python

    Details

      Description

      I am trying current master.

      I ran:

      {{{{
      python setup.py build_ext --build-type=$ARROW_BUILD_TYPE --with-parquet --with-plasma --bundle-arrow-cpp bdist_wheel
      }}}}

      Output:

      {{{{
      running build_ext
      creating build
      creating build/temp.linux-x86_64-3.6
      – Runnning cmake for pyarrow
      cmake -DPYTHON_EXECUTABLE=.../Temp/arrow/pyarrow/bin/python -DPYARROW_BUILD_PARQUET=on -DPYARROW_BOOST_USE_SHARED=on -DPYARROW_BUILD_PLASMA=on -DPYARROW_BUNDLE_ARROW_CPP=ON -DCMAKE_BUILD_TYPE=release .../Temp/arrow/arrow/python
      – The C compiler identification is GNU 7.2.0
      – The CXX compiler identification is GNU 7.2.0
      – Check for working C compiler: /usr/bin/cc
      – Check for working C compiler: /usr/bin/cc – works
      – Detecting C compiler ABI info
      – Detecting C compiler ABI info - done
      – Detecting C compile features
      – Detecting C compile features - done
      – Check for working CXX compiler: /usr/bin/c++
      – Check for working CXX compiler: /usr/bin/c++ – works
      – Detecting CXX compiler ABI info
      – Detecting CXX compiler ABI info - done
      – Detecting CXX compile features
      – Detecting CXX compile features - done
      INFOCompiler command: /usr/bin/c++
      INFOCompiler version: Using built-in specs.
      COLLECT_GCC=/usr/bin/c++
      COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
      OFFLOAD_TARGET_NAMES=nvptx-none
      OFFLOAD_TARGET_DEFAULT=1
      Target: x86_64-linux-gnu
      Configured with: ../src/configure v --with-pkgversion='Ubuntu 7.2.0-8ubuntu3.2' --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c+,go,brig,d,fortran,objc,obj-c+ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --with-sysroot=/ --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
      Thread model: posix
      gcc version 7.2.0 (Ubuntu 7.2.0-8ubuntu3.2)

      INFOCompiler id: GNU
      Selected compiler gcc 7.2.0
      – Performing Test CXX_SUPPORTS_SSE3
      – Performing Test CXX_SUPPORTS_SSE3 - Success
      – Performing Test CXX_SUPPORTS_ALTIVEC
      – Performing Test CXX_SUPPORTS_ALTIVEC - Failed
      Configured for RELEASE build (set with cmake -DCMAKE_BUILD_TYPE=

      {release,debug,...}

      )
      – Build Type: RELEASE
      – Build output directory: .../Temp/arrow/arrow/python/build/temp.linux-x86_64-3.6/release/
      – Found PythonInterp: .../Temp/arrow/pyarrow/bin/python (found version "3.6.3")
      – Searching for Python libs in .../Temp/arrow/pyarrow/lib64;.../Temp/arrow/pyarrow/lib;/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu
      – Looking for python3.6m
      – Found Python lib /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.so
      – Found PythonLibs: /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.so
      – Found NumPy: version "1.14.1" .../Temp/arrow/pyarrow/lib/python3.6/site-packages/numpy/core/include
      – Searching for Python libs in .../Temp/arrow/pyarrow/lib64;.../Temp/arrow/pyarrow/lib;/usr/lib/python3.6/config-3.6m-x86_64-linux-gnu
      – Looking for python3.6m
      – Found Python lib /usr/lib/python3.6/config-3.6m-x86_64-linux-gnu/libpython3.6m.so
      – Found PkgConfig: /usr/bin/pkg-config (found version "0.29.1")
      – Checking for module 'arrow'
      – Found arrow, version 0.9.0-SNAPSHOT
      – Arrow ABI version: 0.0.0
      – Arrow SO version: 0
      – Found the Arrow core library: .../Temp/arrow/dist/lib/libarrow.so
      – Found the Arrow Python library: .../Temp/arrow/dist/lib/libarrow_python.so
      – Boost version: 1.63.0
      – Found the following Boost libraries:
      – system
      – filesystem
      – regex
      Added shared library dependency arrow: .../Temp/arrow/dist/lib/libarrow.so
      Added shared library dependency arrow_python: .../Temp/arrow/dist/lib/libarrow_python.so
      – Found the Parquet library: .../Temp/arrow/dist/lib/libparquet.so
      Added shared library dependency parquet: .../Temp/arrow/dist/lib/libparquet.so
      – Checking for module 'plasma'
      – Found plasma, version
      – Plasma ABI version: 0.0.0
      – Plasma SO version: 0
      – Found the Plasma core library: .../Temp/arrow/dist/lib/libplasma.so
      – Found Plasma executable: .../Temp/arrow/dist/bin/plasma_store
      Added shared library dependency libplasma: .../Temp/arrow/dist/lib/libplasma.so
      – Configuring done
      – Generating done
      – Build files have been written to: .../Temp/arrow/arrow/python/build/temp.linux-x86_64-3.6
      – Finished cmake for pyarrow
      – Running cmake --build for pyarrow
      make
      Scanning dependencies of target lib_pyx
      [ 11%] Compiling Cython CXX source for lib...
      [ 11%] Built target lib_pyx
      Scanning dependencies of target lib
      [ 22%] Building CXX object CMakeFiles/lib.dir/lib.cxx.o
      [ 33%] Linking CXX shared module release/lib.cpython-36m-x86_64-linux-gnu.so
      [ 33%] Built target lib
      Scanning dependencies of target _parquet_pyx
      [ 44%] Compiling Cython CXX source for _parquet...
      [ 44%] Built target _parquet_pyx
      Scanning dependencies of target _parquet
      [ 55%] Building CXX object CMakeFiles/_parquet.dir/_parquet.cxx.o
      [ 66%] Linking CXX shared module release/_parquet.cpython-36m-x86_64-linux-gnu.so
      [ 66%] Built target _parquet
      Scanning dependencies of target plasma_pyx
      [ 77%] Compiling Cython CXX source for plasma...
      [ 77%] Built target plasma_pyx
      Scanning dependencies of target plasma
      [ 88%] Building CXX object CMakeFiles/plasma.dir/plasma.cxx.o
      [100%] Linking CXX shared module release/plasma.cpython-36m-x86_64-linux-gnu.so
      [100%] Built target plasma
      – Finished cmake --build for pyarrow
      .../Temp/arrow/arrow/python/build/lib.linux-x86_64-3.6/pyarrow
      ['release/libarrow.so.0.0.0', 'release/libarrow.so.0', 'release/libarrow.so']
      release/libarrow.so.0.0.0
      ['release/libarrow_python.so.0.0.0', 'release/libarrow_python.so.0', 'release/libarrow_python.so']
      release/libarrow_python.so.0.0.0
      ['release/libplasma.so.0.0.0', 'release/libplasma.so.0', 'release/libplasma.so']
      release/libplasma.so.0.0.0
      ['release/libparquet.so.1.3.2', 'release/libparquet.so.1', 'release/libparquet.so']
      release/libparquet.so.1.3.2
      }}}}

      And after that it crashes because it cannot find arrow_boost_filesystem in build_prefix in _move_shared_libs_unix call which makes libs variable there empty.

      If I do:

      {{{{
      $ find . -name *.so
      ./build/lib.linux-x86_64-3.6/pyarrow/libparquet.so
      ./build/lib.linux-x86_64-3.6/pyarrow/libplasma.so
      ./build/lib.linux-x86_64-3.6/pyarrow/libarrow.so
      ./build/lib.linux-x86_64-3.6/pyarrow/libarrow_python.so
      ./build/temp.linux-x86_64-3.6/release/_parquet.cpython-36m-x86_64-linux-gnu.so
      ./build/temp.linux-x86_64-3.6/release/libparquet.so
      ./build/temp.linux-x86_64-3.6/release/libboost_filesystem.so
      ./build/temp.linux-x86_64-3.6/release/libplasma.so
      ./build/temp.linux-x86_64-3.6/release/libarrow.so
      ./build/temp.linux-x86_64-3.6/release/libboost_system.so
      ./build/temp.linux-x86_64-3.6/release/lib.cpython-36m-x86_64-linux-gnu.so
      ./build/temp.linux-x86_64-3.6/release/libboost_regex.so
      ./build/temp.linux-x86_64-3.6/release/libarrow_python.so
      ./build/temp.linux-x86_64-3.6/release/plasma.cpython-36m-x86_64-linux-gnu.so
      ./pyarrow/_parquet.cpython-36m-x86_64-linux-gnu.so
      ./pyarrow/lib.cpython-36m-x86_64-linux-gnu.so
      ./pyarrow/plasma.cpython-36m-x86_64-linux-gnu.so
      }}}}

      So it seems the issue is that the filename is libboost_filesystem.so, but it is trying to move libarrow_boost_filesystem.so?

      Changing lines in setup.py to:

      {{{{
      move_shared_libs(build_prefix, build_lib, "boost_filesystem")
      move_shared_libs(build_prefix, build_lib, "boost_system")
      move_shared_libs(build_prefix, build_lib, "boost_regex")
      }}}}

      seems to make it build.

        Attachments

          Activity

            People

            • Assignee:
              xhochy Uwe L. Korn
              Reporter:
              mitar Mitar
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Time Tracking

                Estimated:
                Original Estimate - Not Specified
                Not Specified
                Remaining:
                Remaining Estimate - 0h
                0h
                Logged:
                Time Spent - 20m
                20m