Uploaded image for project: 'Apache Arrow'
  1. Apache Arrow
  2. ARROW-4935

[C++] Errors from jemalloc when building pyarrow from source on OSX and Debian

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Not A Problem
    • Affects Version/s: 0.12.1
    • Fix Version/s: None
    • Component/s: C++, Python
    • Labels:
    • Environment:
      OSX, Debian, Python==3.6.7

      Description

      My attempts to build pyarrow from source are failing. I've set up the conda environment using the instructions provided in the Develop instructions, and have tried this on both Debian and OSX. When I run CMAKE in debug mode on OSX, the output is:

      -- Building using CMake version: 3.14.0
      -- Arrow version: 0.13.0 (full: '0.13.0-SNAPSHOT')
      -- clang-tidy not found
      -- clang-format not found
      -- infer found at /usr/local/bin/infer
      -- Using ccache: /usr/local/bin/ccache
      -- Found cpplint executable at /Users/Greg/documents/repos/arrow/cpp/build-support/cpplint.py
      -- Compiler command: env LANG=C /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/c++ -v
      -- Compiler version: Apple LLVM version 10.0.0 (clang-1000.11.45.5)
      Target: x86_64-apple-darwin18.2.0
      Thread model: posix
      InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin
      
      -- Compiler id: AppleClang
      Selected compiler clang 4.1.0svn
      -- Arrow build warning level: CHECKIN
      Configured for DEBUG build (set with cmake -DCMAKE_BUILD_TYPE={release,debug,...})
      -- Build Type: DEBUG
      -- BOOST_VERSION: 1.67.0
      -- BROTLI_VERSION: v0.6.0
      -- CARES_VERSION: 1.15.0
      -- DOUBLE_CONVERSION_VERSION: v3.1.1
      -- FLATBUFFERS_VERSION: v1.10.0
      -- GBENCHMARK_VERSION: v1.4.1
      -- GFLAGS_VERSION: v2.2.0
      -- GLOG_VERSION: v0.3.5
      -- GRPC_VERSION: v1.18.0
      -- GTEST_VERSION: 1.8.1
      -- JEMALLOC_VERSION: 17c897976c60b0e6e4f4a365c751027244dada7a
      -- LZ4_VERSION: v1.8.3
      -- ORC_VERSION: 1.5.4
      -- PROTOBUF_VERSION: v3.6.1
      -- RAPIDJSON_VERSION: v1.1.0
      -- RE2_VERSION: 2018-10-01
      -- SNAPPY_VERSION: 1.1.3
      -- THRIFT_VERSION: 0.11.0
      -- ZLIB_VERSION: 1.2.8
      -- ZSTD_VERSION: v1.3.7
      -- Boost version: 1.68.0
      -- Found the following Boost libraries:
      --   regex
      --   system
      --   filesystem
      -- Boost include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
      -- Boost libraries: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_regex.dylib/Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_system.dylib/Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_filesystem.dylib
      Added shared library dependency boost_system_shared: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_system.dylib
      Added shared library dependency boost_filesystem_shared: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_filesystem.dylib
      Added shared library dependency boost_regex_shared: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libboost_regex.dylib
      Added static library dependency double-conversion_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libdouble-conversion.a
      -- double-conversion include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
      -- double-conversion static library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libdouble-conversion.a
      -- GFLAGS_HOME: /Users/Greg/anaconda3/envs/pyarrow-dev
      -- GFlags include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
      -- GFlags static library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libgflags.a
      Added static library dependency gflags_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libgflags.a
      -- RapidJSON include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
      -- Found the Flatbuffers library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libflatbuffers.a
      -- Flatbuffers include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
      -- Flatbuffers compiler: /Users/Greg/anaconda3/envs/pyarrow-dev/bin/flatc
      Added static library dependency jemalloc_static: /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist//lib/libjemalloc_pic.a
      Added shared library dependency jemalloc_shared: /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist//lib/libjemalloc.dylib
      -- Found hdfs.h at: /Users/Greg/documents/repos/arrow/cpp/thirdparty/hadoop/include/hdfs.h
      -- Found the ZLIB shared library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libz.dylib
      Added shared library dependency zlib_shared: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libz.dylib
      -- SNAPPY_HOME: /Users/Greg/anaconda3/envs/pyarrow-dev
      -- Found the Snappy library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libsnappy.dylib
      Added static library dependency snappy_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libsnappy.a
      -- Found the Brotli library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlienc-static.a;/Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlidec-static.a;/Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlicommon-static.a
      Added static library dependency brotli_enc_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlienc-static.a
      Added static library dependency brotli_dec_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlidec-static.a
      Added static library dependency brotli_common_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libbrotlicommon-static.a
      Added static library dependency lz4_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/liblz4.a
      Added static library dependency zstd_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libzstd.a
      -- THRIFT_HOME: /Users/Greg/anaconda3/envs/pyarrow-dev
      -- Thrift version: 0.12.0
      -- Thrift include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
      -- Thrift static library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libthrift.a
      -- Thrift compiler: /Users/Greg/anaconda3/envs/pyarrow-dev/bin/thrift
      -- GLOG_HOME: /Users/Greg/anaconda3/envs/pyarrow-dev
      -- Found the GLog library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libglog.dylib
      -- Glog include dir: /Users/Greg/anaconda3/envs/pyarrow-dev/include
      -- Glog static library: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libglog.a
      Added static library dependency glog_static: /Users/Greg/anaconda3/envs/pyarrow-dev/lib/libglog.a
      -- CMAKE_C_FLAGS:  -Qunused-arguments -ggdb -O0   -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-deprecated -Wno-weak-vtables -Wno-padded -Wno-comma -Wno-unused-macros -Wno-unused-parameter -Wno-unused-template -Wno-undef -Wno-shadow -Wno-switch-enum -Wno-exit-time-destructors -Wno-global-constructors -Wno-weak-template-vtables -Wno-undefined-reinterpret-cast -Wno-implicit-fallthrough -Wno-unreachable-code-return -Wno-float-equal -Wno-missing-prototypes -Wno-documentation-unknown-command -Wno-old-style-cast -Wno-covered-switch-default -Wno-cast-align -Wno-vla-extension -Wno-shift-sign-overflow -Wno-used-but-marked-unused -Wno-missing-variable-declarations -Wno-gnu-zero-variadic-macro-arguments -Wconversion -Wno-sign-conversion -Wno-disabled-macro-expansion -Wno-format-nonliteral -Wno-missing-noreturn -Wno-gnu-folding-constant -Wno-reserved-id-macro -Wno-range-loop-analysis -Wno-double-promotion -Wno-undefined-func-template -Wno-zero-as-null-pointer-constant -Wno-unknown-warning-option -Werror -Wno-unknown-warning-option -msse4.2 -maltivec -stdlib=libc++
      -- CMAKE_CXX_FLAGS:   -Qunused-arguments -ggdb -O0  -Weverything -Wno-c++98-compat -Wno-c++98-compat-pedantic -Wno-deprecated -Wno-weak-vtables -Wno-padded -Wno-comma -Wno-unused-macros -Wno-unused-parameter -Wno-unused-template -Wno-undef -Wno-shadow -Wno-switch-enum -Wno-exit-time-destructors -Wno-global-constructors -Wno-weak-template-vtables -Wno-undefined-reinterpret-cast -Wno-implicit-fallthrough -Wno-unreachable-code-return -Wno-float-equal -Wno-missing-prototypes -Wno-documentation-unknown-command -Wno-old-style-cast -Wno-covered-switch-default -Wno-cast-align -Wno-vla-extension -Wno-shift-sign-overflow -Wno-used-but-marked-unused -Wno-missing-variable-declarations -Wno-gnu-zero-variadic-macro-arguments -Wconversion -Wno-sign-conversion -Wno-disabled-macro-expansion -Wno-format-nonliteral -Wno-missing-noreturn -Wno-gnu-folding-constant -Wno-reserved-id-macro -Wno-range-loop-analysis -Wno-double-promotion -Wno-undefined-func-template -Wno-zero-as-null-pointer-constant -Wno-unknown-warning-option -Werror -Wno-unknown-warning-option -msse4.2 -maltivec -stdlib=libc++ 
      
      
      -- Configuring done
      CMake Warning (dev):
        Policy CMP0068 is not set: RPATH settings on macOS do not affect
        install_name.  Run "cmake --help-policy CMP0068" for policy details.  Use
        the cmake_policy command to set the policy and suppress this warning.
      
        For compatibility with older versions of CMake, the install_name fields for
        the following targets are still affected by RPATH settings:
      
         arrow_python_shared
         arrow_shared
         parquet_shared
         plasma_shared
      
      This warning is for project developers.  Use -Wno-dev to suppress it.
      
      -- Generating done
      -- Build files have been written to: /Users/Greg/documents/repos/arrow/cpp/build

      After I run make -j4, the output is:

        0%] Built target metadata_fbs
      [  1%] Performing configure step for 'jemalloc_ep'
      [  2%] Built target parquet-thrift-deps
      [ 12%] Built target arrow_python_objlib
      [ 19%] Built target plasma_objlib
      [ 19%] Built target arrow_python_static
      CMake Error at /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-configure-DEBUG.cmake:49 (message):
        Command failed: 1
      
         './autogen.sh' 'AR=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar' 'CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc' '--prefix=/Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist/' '--with-jemalloc-prefix=je_arrow_' '--with-private-namespace=je_arrow_private_' '--disable-tls'
      
        See also
      
          /Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-configure-*.log
      
      
      make[2]: *** [jemalloc_ep-prefix/src/jemalloc_ep-stamp/jemalloc_ep-configure] Error 1
      make[1]: *** [CMakeFiles/jemalloc_ep.dir/all] Error 2
      make: *** [all] Error 2
      

      And the output of the jemalloc log file is:

      configure: error: in `/Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep':
      configure: error: cannot run C compiled programs.
      If you meant to cross compile, use `--host'.
      See `config.log' for more details
      autoconf
      ./configure --enable-autogen AR=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/ar CC=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc --prefix=/Users/Greg/documents/repos/arrow/cpp/build/jemalloc_ep-prefix/src/jemalloc_ep/dist/ --with-jemalloc-prefix=je_arrow_ --with-private-namespace=je_arrow_private_ --disable-tls
      checking for xsltproc... /usr/bin/xsltproc
      checking for gcc... /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin/cc
      checking whether the C compiler works... yes
      checking for C compiler default output file name... a.out
      checking for suffix of executables... 
      checking whether we are cross compiling... Error 0 in ./configure
      

      Any help / guidance I can get is appreciated.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              hayesgb Gregory Hayes
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: