Hadoop Common
  1. Hadoop Common
  2. HADOOP-8659

Native libraries must build with soft-float ABI for Oracle JVM on ARM

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.2-alpha
    • Component/s: native
    • Labels:
      None
    • Environment:

      armhf Linux with Oracle JVM

    • Hadoop Flags:
      Reviewed

      Description

      There was recently an ABI (application binary interface) change in most Linux distributions for modern ARM processors (ARMv7). Historically, hardware floating-point (FP) support was optional/vendor-specific for ARM processors, so for software compatibility, the default ABI required that processors with FP units copy FP arguments into integer registers (or memory) when calling a shared library function. Now that hardware floating-point has been standardized for some time, Linux distributions such as Ubuntu 12.04 have changed the default ABI to leave FP arguments in FP registers, since this can significantly improve performance for FP libraries.

      Unfortunately, Oracle has not yet released a JVM (as of 7u4) that supports the new ABI, presumably since this involves some non-trivial changes to components like JNI. While the soft-float JVM can run on systems with multi-arch support (currently Debian/Ubuntu) using compatibility libraries, this configuration requires that any third-party JNI libraries also be compiled using the soft-float ABI. Since hard-float systems default to compiling for hard-float, an extra argument to GCC (and installation of a compatibility library) is required to build soft-float Hadoop native libraries that work with the Oracle JVM.

      Note that OpenJDK on hard-float systems does use the new ABI, and expects JNI libraries to use it as well. Therefore the fix for this issue requires detecting the float ABI of the current JVM.

      1. HADOOP-8659-fix-002.patch
        1 kB
        Colin Patrick McCabe
      2. HADOOP-8659-fix-001.patch
        1 kB
        Colin Patrick McCabe
      3. HADOOP-8659.patch
        8 kB
        Trevor Robinson
      4. HADOOP-8659.patch
        7 kB
        Trevor Robinson
      5. HADOOP-8659.patch
        7 kB
        Trevor Robinson
      6. HADOOP-8659.patch
        7 kB
        Trevor Robinson

        Activity

        Hide
        Trevor Robinson added a comment -

        The attached patch factors out platform-specific build configuration for various native libraries (e.g. HADOOP-8538) into a single included file and adds support for building soft-float libraries on hard-float ARM systems when using a soft-float JVM.

        Show
        Trevor Robinson added a comment - The attached patch factors out platform-specific build configuration for various native libraries (e.g. HADOOP-8538 ) into a single included file and adds support for building soft-float libraries on hard-float ARM systems when using a soft-float JVM.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12539758/HADOOP-8659.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1262//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539758/HADOOP-8659.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1262//console This message is automatically generated.
        Hide
        Todd Lipcon added a comment -

        I don't know the cmake stuff quite well enough to review, but one question: why does this affect us despite not having any calls in libhadoop that use float arguments? Is the calling convention different even for non-float args?

        Show
        Todd Lipcon added a comment - I don't know the cmake stuff quite well enough to review, but one question: why does this affect us despite not having any calls in libhadoop that use float arguments? Is the calling convention different even for non-float args?
        Hide
        Trevor Robinson added a comment -

        I don't think it's different for non-float args. The problem is all of the transitive dependencies, such as using a different libc. When trying to load a JNI native library with the wrong float ABI, the JVM usually crashes silently with exit code 1. For instance, the build currently dies on hard-float ARM with the Oracle JVM running hadoop-hdfs-project/hadoop-hdfs/target/native/test_libhdfs_threaded.

        Show
        Trevor Robinson added a comment - I don't think it's different for non-float args. The problem is all of the transitive dependencies, such as using a different libc. When trying to load a JNI native library with the wrong float ABI, the JVM usually crashes silently with exit code 1. For instance, the build currently dies on hard-float ARM with the Oracle JVM running hadoop-hdfs-project/hadoop-hdfs/target/native/test_libhdfs_threaded.
        Hide
        Trevor Robinson added a comment -

        Updated patch based on testing with hard-float OpenJDK. Also verified unchanged behavior on x86-64.

        Show
        Trevor Robinson added a comment - Updated patch based on testing with hard-float OpenJDK. Also verified unchanged behavior on x86-64.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12539768/HADOOP-8659.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1263//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539768/HADOOP-8659.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac. The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1263//console This message is automatically generated.
        Hide
        Trevor Robinson added a comment -

        Update patch to remove unnecessary dependency on JAVA_JVM_LIBRARY from hadooppipes, which caused build failure in Jenkins:

        CMake Error: The following variables are used in this project, but they are set to NOTFOUND.
        Please set them or make sure they are set and tested correctly in the CMake files:
        JAVA_JVM_LIBRARY (ADVANCED)
            linked by target "hadooppipes" in directory /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-tools/hadoop-pipes/src
        
        Show
        Trevor Robinson added a comment - Update patch to remove unnecessary dependency on JAVA_JVM_LIBRARY from hadooppipes, which caused build failure in Jenkins: CMake Error: The following variables are used in this project, but they are set to NOTFOUND. Please set them or make sure they are set and tested correctly in the CMake files: JAVA_JVM_LIBRARY (ADVANCED) linked by target "hadooppipes" in directory /home/jenkins/jenkins-slave/workspace/PreCommit-HADOOP-Build/trunk/hadoop-tools/hadoop-pipes/src
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12539913/HADOOP-8659.patch
        against trunk revision .

        -1 patch. The patch command could not apply the patch.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1269//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539913/HADOOP-8659.patch against trunk revision . -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1269//console This message is automatically generated.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12539957/HADOOP-8659.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

        org.apache.hadoop.ha.TestZKFailoverController
        org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS
        org.apache.hadoop.hdfs.TestFileConcurrentReader
        org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics
        org.apache.hadoop.hdfs.server.namenode.TestFsck

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1271//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1271//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12539957/HADOOP-8659.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager: org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS org.apache.hadoop.hdfs.TestFileConcurrentReader org.apache.hadoop.hdfs.server.namenode.metrics.TestNameNodeMetrics org.apache.hadoop.hdfs.server.namenode.TestFsck +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1271//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1271//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -

        This looks good overall.

        +    else ()
        +        # On hard-float systems, soft-float compatibility dev packages are required,
        +        # e.g. libc6-dev-armel on Ubuntu 12.04.
        +        message("Soft-float JVM detected; ensure that soft-float dev packages are installed")
        +        set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mfloat-abi=softfp")
        +    endif ()
        

        Would it be possible to use CHECK_SYMBOL_EXISTS or CHECK_LIBRARY_EXISTS to ensure that the soft-float dev packages are installed? I'm not too familiar with soft-float libraries on ARM, so I'm just guessing here.

        +    execute_process(
        +        COMMAND readelf -A ${JAVA_JVM_LIBRARY}
        +        OUTPUT_VARIABLE JVM_ELF_ARCH
        +        ERROR_QUIET)
        +    if (JVM_ELF_ARCH MATCHES "Tag_ABI_VFP_args: VFP registers")
        +        message("Hard-float JVM detected")
        +        set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mfloat-abi=hard")
        

        I wish there were a way to use CHECK_C_SOURCE_COMPILES or something here to determine if the JVM library was soft-float or hard-float. I don't know if everyone has readelf installed by default, and it's preferable to reduce the number of dependencies we have. However, if you can't find an easy way to do this, then feel free to ignore this comment.

        Show
        Colin Patrick McCabe added a comment - This looks good overall. + else () + # On hard- float systems, soft- float compatibility dev packages are required, + # e.g. libc6-dev-armel on Ubuntu 12.04. + message( "Soft- float JVM detected; ensure that soft- float dev packages are installed" ) + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mfloat-abi=softfp" ) + endif () Would it be possible to use CHECK_SYMBOL_EXISTS or CHECK_LIBRARY_EXISTS to ensure that the soft-float dev packages are installed? I'm not too familiar with soft-float libraries on ARM, so I'm just guessing here. + execute_process( + COMMAND readelf -A ${JAVA_JVM_LIBRARY} + OUTPUT_VARIABLE JVM_ELF_ARCH + ERROR_QUIET) + if (JVM_ELF_ARCH MATCHES "Tag_ABI_VFP_args: VFP registers" ) + message( "Hard- float JVM detected" ) + set(CMAKE_C_FLAGS "${CMAKE_C_FLAGS} -mfloat-abi=hard" ) I wish there were a way to use CHECK_C_SOURCE_COMPILES or something here to determine if the JVM library was soft-float or hard-float. I don't know if everyone has readelf installed by default, and it's preferable to reduce the number of dependencies we have. However, if you can't find an easy way to do this, then feel free to ignore this comment.
        Hide
        Trevor Robinson added a comment -

        It's not as easy as just CHECK_SYMBOL_EXISTS/CHECK_LIBRARY_EXISTS, since the soft-float libraries are identical to the hard-float ones, but are installed in different directories. However, I can do a test compilation against an arbitrary libc symbol with the soft-float flag:

        include(CMakePushCheckState)
        cmake_push_check_state()
        set(CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS} -mfloat-abi=softfp)
        include(CheckSymbolExists)
        check_symbol_exists(exit stdlib.h SOFTFP_AVAILABLE)
        cmake_pop_check_state()
        

        Unfortunately, there is currently no good way to determine the JVM's float ABI. It's not reported at all by the Oracle EJRE or OpenJDK. The current behavior of linking against the JVM library with the wrong ABI doesn't report an error. What I can do is restrict this code path to Linux (since this issue is Linux-specific for now), where readelf is part of binutils (like ld), so it should always be available. But I'll also check for it and issue a warning if it's not found. For example:

        find_program(READELF readelf)
        if (READELF MATCHES "NOTFOUND")
            message(WARNING "readelf not found; JVM float ABI detection disabled")
        endif ()
        
        Show
        Trevor Robinson added a comment - It's not as easy as just CHECK_SYMBOL_EXISTS/CHECK_LIBRARY_EXISTS, since the soft-float libraries are identical to the hard-float ones, but are installed in different directories. However, I can do a test compilation against an arbitrary libc symbol with the soft-float flag: include(CMakePushCheckState) cmake_push_check_state() set(CMAKE_REQUIRED_FLAGS ${CMAKE_REQUIRED_FLAGS} -mfloat-abi=softfp) include(CheckSymbolExists) check_symbol_exists(exit stdlib.h SOFTFP_AVAILABLE) cmake_pop_check_state() Unfortunately, there is currently no good way to determine the JVM's float ABI. It's not reported at all by the Oracle EJRE or OpenJDK. The current behavior of linking against the JVM library with the wrong ABI doesn't report an error. What I can do is restrict this code path to Linux (since this issue is Linux-specific for now), where readelf is part of binutils (like ld), so it should always be available. But I'll also check for it and issue a warning if it's not found. For example: find_program(READELF readelf) if (READELF MATCHES "NOTFOUND" ) message(WARNING "readelf not found; JVM float ABI detection disabled" ) endif ()
        Hide
        Trevor Robinson added a comment -

        Attached updated patch based on Colin's comments.

        Show
        Trevor Robinson added a comment - Attached updated patch based on Colin's comments.
        Hide
        Colin Patrick McCabe added a comment -

        Unfortunately, there is currently no good way to determine the JVM's float ABI...

        Yeah, I was afraid of that. That aborting at runtime behavior is really nasty. I'm glad your change prevents people from being exposed to that.

        +1.

        Show
        Colin Patrick McCabe added a comment - Unfortunately, there is currently no good way to determine the JVM's float ABI... Yeah, I was afraid of that. That aborting at runtime behavior is really nasty. I'm glad your change prevents people from being exposed to that. +1.
        Hide
        Todd Lipcon added a comment -

        +1, looks good to me too. Thanks, Trevor.

        Show
        Todd Lipcon added a comment - +1, looks good to me too. Thanks, Trevor.
        Hide
        Todd Lipcon added a comment -

        Committed to trunk and branch-2, thanks Trevor!

        Show
        Todd Lipcon added a comment - Committed to trunk and branch-2, thanks Trevor!
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2634 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2634/)
        HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM on ARM. Contributed by Trevor Robinson. (Revision 1371507)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371507
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2634 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2634/ ) HADOOP-8659 . Native libraries must build with soft-float ABI for Oracle JVM on ARM. Contributed by Trevor Robinson. (Revision 1371507) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371507 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2569 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2569/)
        HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM on ARM. Contributed by Trevor Robinson. (Revision 1371507)

        Result = SUCCESS
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371507
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2569 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2569/ ) HADOOP-8659 . Native libraries must build with soft-float ABI for Oracle JVM on ARM. Contributed by Trevor Robinson. (Revision 1371507) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371507 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2590 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2590/)
        HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM on ARM. Contributed by Trevor Robinson. (Revision 1371507)

        Result = FAILURE
        todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371507
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        • /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt
        • /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2590 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2590/ ) HADOOP-8659 . Native libraries must build with soft-float ABI for Oracle JVM on ARM. Contributed by Trevor Robinson. (Revision 1371507) Result = FAILURE todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1371507 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/CMakeLists.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/CMakeLists.txt /hadoop/common/trunk/hadoop-tools/hadoop-pipes/src/CMakeLists.txt /hadoop/common/trunk/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
        Hide
        Tsz Wo Nicholas Sze added a comment -
        Show
        Tsz Wo Nicholas Sze added a comment - This might break the builds in Jenkins. Please take a look. https://builds.apache.org/job/PreCommit-HDFS-Build/2988/artifact/trunk/patchprocess/patchJavacWarnings.txt
        Hide
        Trevor Robinson added a comment -

        What's the difference between https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/ and https://builds.apache.org/job/PreCommit-HDFS-Build/? The former is passing but the latter is failing. Does the former not build native libraries? Also https://builds.apache.org/job/PreCommit-HDFS-Build/2983/ included this change but appears to have built successfully. I'm baffled right now, but it's also past 3am for me.

        Show
        Trevor Robinson added a comment - What's the difference between https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/ and https://builds.apache.org/job/PreCommit-HDFS-Build/? The former is passing but the latter is failing. Does the former not build native libraries? Also https://builds.apache.org/job/PreCommit-HDFS-Build/2983/ included this change but appears to have built successfully. I'm baffled right now, but it's also past 3am for me.
        Hide
        Colin Patrick McCabe added a comment -

        I think this patch will fix it... let me run it past Jenkins

        Show
        Colin Patrick McCabe added a comment - I think this patch will fix it... let me run it past Jenkins
        Hide
        Colin Patrick McCabe added a comment -

        So the build issue here is basically a re-introduction of HADOOP-8489.

        The issue is that when you do a 32-bit compile on a 64-bit machine, with both 32 and 64-bit JVM libraries in your path, you need to make sure you choose the 32-bit JVM libraries. The way we do this is by setting CMAKE_SYSTEM_PROCESOR. However, you must do this before find_package(JNI REQUIRED); otherwise, the 64-bit libraries will be found and used, which results in a linker error when you try to link them with the code which was compiled with -m32.

        Show
        Colin Patrick McCabe added a comment - So the build issue here is basically a re-introduction of HADOOP-8489 . The issue is that when you do a 32-bit compile on a 64-bit machine, with both 32 and 64-bit JVM libraries in your path, you need to make sure you choose the 32-bit JVM libraries. The way we do this is by setting CMAKE_SYSTEM_PROCESOR . However, you must do this before find_package(JNI REQUIRED) ; otherwise, the 64-bit libraries will be found and used, which results in a linker error when you try to link them with the code which was compiled with -m32 .
        Hide
        Colin Patrick McCabe added a comment -

        should add -m32 to CMAKE_CXX_FLAGS as well.

        Show
        Colin Patrick McCabe added a comment - should add -m32 to CMAKE_CXX_FLAGS as well.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12540690/HADOOP-8659-fix-001.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.ha.TestZKFailoverController

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1281//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1281//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540690/HADOOP-8659-fix-001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1281//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1281//console This message is automatically generated.
        Hide
        Trevor Robinson added a comment -

        Thanks for fixing this, Colin.

        we do this is by setting CMAKE_SYSTEM_PROCESOR. However, you must do this before find_package(JNI REQUIRED)

        So that's why CMAKE_SYSTEM_PROCESSOR was being set... This subtlety screams for a comment in the code.

        Show
        Trevor Robinson added a comment - Thanks for fixing this, Colin. we do this is by setting CMAKE_SYSTEM_PROCESOR. However, you must do this before find_package(JNI REQUIRED) So that's why CMAKE_SYSTEM_PROCESSOR was being set... This subtlety screams for a comment in the code.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12540712/HADOOP-8659-fix-001.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.ha.TestZKFailoverController

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1282//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1282//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540712/HADOOP-8659-fix-001.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.ha.TestZKFailoverController +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1282//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1282//console This message is automatically generated.
        Hide
        Colin Patrick McCabe added a comment -
        • add a comment about CMAKE_SYSTEM_PROCESSOR
        Show
        Colin Patrick McCabe added a comment - add a comment about CMAKE_SYSTEM_PROCESSOR
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12540736/HADOOP-8659-fix-002.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. The javadoc tool did not generate any warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1285//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1285//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12540736/HADOOP-8659-fix-002.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 javadoc. The javadoc tool did not generate any warning messages. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common. +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/1285//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/1285//console This message is automatically generated.
        Hide
        Eli Collins added a comment -

        +1 to the latest fix. Colin, in the future please file a new jira for issues like this.

        Show
        Eli Collins added a comment - +1 to the latest fix. Colin, in the future please file a new jira for issues like this.
        Hide
        Eli Collins added a comment -

        I've committed the fix and merged to branch-2.

        Show
        Eli Collins added a comment - I've committed the fix and merged to branch-2.
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk-Commit #2595 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2595/)
        Amend HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk-Commit #2595 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2595/ ) Amend HADOOP-8659 . Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk-Commit #2639 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2639/)
        Amend HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk-Commit #2639 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2639/ ) Amend HADOOP-8659 . Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Common-trunk-Commit #2574 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2574/)
        Amend HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583)

        Result = SUCCESS
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - Integrated in Hadoop-Common-trunk-Commit #2574 (See https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2574/ ) Amend HADOOP-8659 . Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/)
        Amend HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - Integrated in Hadoop-Hdfs-trunk #1135 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1135/ ) Amend HADOOP-8659 . Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Hide
        Hudson added a comment -

        Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/)
        Amend HADOOP-8659. Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583)

        Result = FAILURE
        eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583
        Files :

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - Integrated in Hadoop-Mapreduce-trunk #1167 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1167/ ) Amend HADOOP-8659 . Native libraries must build with soft-float ABI for Oracle JVM on ARM. (Revision 1372583) Result = FAILURE eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1372583 Files : /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake

          People

          • Assignee:
            Colin Patrick McCabe
            Reporter:
            Trevor Robinson
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development