Hadoop Common
  1. Hadoop Common
  2. HADOOP-10968

hadoop native build fails to detect java_libarch on ppc64le

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.3.0
    • Fix Version/s: 2.6.0
    • Component/s: build
    • Labels:
      None
    • Target Version/s:

      Description

      [INFO]
      [INFO] — maven-antrun-plugin:1.7:run (make) @ hadoop-common —
      [INFO] Executing tasks

      main:
      [exec] – The C compiler identification is GNU 4.8.3
      [exec] – The CXX compiler identification is GNU 4.8.3
      [exec] – Check for working C compiler: /usr/bin/cc
      [exec] – Check for working C compiler: /usr/bin/cc – works
      [exec] – Detecting C compiler ABI info
      [exec] – Detecting C compiler ABI info - done
      [exec] – Check for working CXX compiler: /usr/bin/c++
      [exec] – Check for working CXX compiler: /usr/bin/c++ – works
      [exec] JAVA_HOME=, JAVA_JVM_LIBRARY=JAVA_JVM_LIBRARY-NOTFOUND
      [exec] JAVA_INCLUDE_PATH=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include, JAVA_INCLUDE_PATH2=/usr/lib64/jvm/java-1.7.0-openjdk-1.7.0/include/linux
      [exec] CMake Error at JNIFlags.cmake:114 (MESSAGE):
      [exec] Failed to find a viable JVM installation under JAVA_HOME.
      [exec] Call Stack (most recent call first):
      [exec] CMakeLists.txt:24 (include)
      [exec]
      [exec]
      [exec] – Detecting CXX compiler ABI info
      [exec] – Detecting CXX compiler ABI info - done
      [exec] – Configuring incomplete, errors occurred!
      [exec] See also "/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native/CMakeFiles/CMakeOutput.log".
      [INFO] ------------------------------------------------------------------------
      [INFO] Reactor Summary:
      [INFO]
      [INFO] Apache Hadoop Main ................................. SUCCESS [ 10.680 s]
      [INFO] Apache Hadoop Project POM .......................... SUCCESS [ 0.716 s]
      [INFO] Apache Hadoop Annotations .......................... SUCCESS [ 3.270 s]
      [INFO] Apache Hadoop Assemblies ........................... SUCCESS [ 0.274 s]
      [INFO] Apache Hadoop Project Dist POM ..................... SUCCESS [ 1.819 s]
      [INFO] Apache Hadoop Maven Plugins ........................ SUCCESS [ 3.284 s]
      [INFO] Apache Hadoop MiniKDC .............................. SUCCESS [ 2.863 s]
      [INFO] Apache Hadoop Auth ................................. SUCCESS [ 4.032 s]
      [INFO] Apache Hadoop Auth Examples ........................ SUCCESS [ 2.475 s]
      [INFO] Apache Hadoop Common ............................... FAILURE [ 10.458 s]
      [INFO] Apache Hadoop NFS .................................. SKIPPED
      [INFO] Apache Hadoop Common Project ....................... SKIPPED
      [INFO] Apache Hadoop HDFS ................................. SKIPPED
      [INFO] Apache Hadoop HttpFS ............................... SKIPPED
      [INFO] Apache Hadoop HDFS BookKeeper Journal .............. SKIPPED
      [INFO] Apache Hadoop HDFS-NFS ............................. SKIPPED
      [INFO] Apache Hadoop HDFS Project ......................... SKIPPED
      [INFO] hadoop-yarn ........................................ SKIPPED
      [INFO] hadoop-yarn-api .................................... SKIPPED
      [INFO] hadoop-yarn-common ................................. SKIPPED
      [INFO] hadoop-yarn-server ................................. SKIPPED
      [INFO] hadoop-yarn-server-common .......................... SKIPPED
      [INFO] hadoop-yarn-server-nodemanager ..................... SKIPPED
      [INFO] hadoop-yarn-server-web-proxy ....................... SKIPPED
      [INFO] hadoop-yarn-server-resourcemanager ................. SKIPPED
      [INFO] hadoop-yarn-server-tests ........................... SKIPPED
      [INFO] hadoop-yarn-client ................................. SKIPPED
      [INFO] hadoop-yarn-applications ........................... SKIPPED
      [INFO] hadoop-yarn-applications-distributedshell .......... SKIPPED
      [INFO] hadoop-yarn-applications-unmanaged-am-launcher ..... SKIPPED
      [INFO] hadoop-yarn-site ................................... SKIPPED
      [INFO] hadoop-yarn-project ................................ SKIPPED
      [INFO] hadoop-mapreduce-client ............................ SKIPPED
      [INFO] hadoop-mapreduce-client-core ....................... SKIPPED
      [INFO] hadoop-mapreduce-client-common ..................... SKIPPED
      [INFO] hadoop-mapreduce-client-shuffle .................... SKIPPED
      [INFO] hadoop-mapreduce-client-app ........................ SKIPPED
      [INFO] hadoop-mapreduce-client-hs ......................... SKIPPED
      [INFO] hadoop-mapreduce-client-jobclient .................. SKIPPED
      [INFO] hadoop-mapreduce-client-hs-plugins ................. SKIPPED
      [INFO] Apache Hadoop MapReduce Examples ................... SKIPPED
      [INFO] hadoop-mapreduce ................................... SKIPPED
      [INFO] Apache Hadoop MapReduce Streaming .................. SKIPPED
      [INFO] Apache Hadoop Distributed Copy ..................... SKIPPED
      [INFO] Apache Hadoop Archives ............................. SKIPPED
      [INFO] Apache Hadoop Rumen ................................ SKIPPED
      [INFO] Apache Hadoop Gridmix .............................. SKIPPED
      [INFO] Apache Hadoop Data Join ............................ SKIPPED
      [INFO] Apache Hadoop Extras ............................... SKIPPED
      [INFO] Apache Hadoop Pipes ................................ SKIPPED
      [INFO] Apache Hadoop OpenStack support .................... SKIPPED
      [INFO] Apache Hadoop Client ............................... SKIPPED
      [INFO] Apache Hadoop Mini-Cluster ......................... SKIPPED
      [INFO] Apache Hadoop Scheduler Load Simulator ............. SKIPPED
      [INFO] Apache Hadoop Tools Dist ........................... SKIPPED
      [INFO] Apache Hadoop Tools ................................ SKIPPED
      [INFO] Apache Hadoop Distribution ......................... SKIPPED
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------
      [INFO] Total time: 42.273 s
      [INFO] Finished at: 2014-08-13T00:43:06+02:00
      [INFO] Final Memory: 61M/465M
      [INFO] ------------------------------------------------------------------------
      [ERROR] Failed to execute goal org.apache.maven.plugins:maven-antrun-plugin:1.7:run (make) on project hadoop-common: An Ant BuildException has occured: exec returned: 1
      [ERROR] around Ant part ...<exec dir="/root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/native" executable="cmake" failonerror="true">... @ 4:156 in /root/bigtop/build/hadoop/rpm/BUILD/hadoop-2.3.0-src/hadoop-common-project/hadoop-common/target/antrun/build-main.xml
      [ERROR] -> [Help 1]
      [ERROR]
      [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
      [ERROR] Re-run Maven using the -X switch to enable full debug logging.
      [ERROR]
      [ERROR] For more information about the errors and possible solutions, please read the following articles:
      [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
      [ERROR]
      [ERROR] After correcting the problems, you can resume the build with the command
      [ERROR] mvn <goals> -rf :hadoop-common
      error: Bad exit status from /var/tmp/rpm-tmp.suXMUs (%build)

        Activity

        Hide
        Tony Reix added a comment -

        The 2 versions that shown issue with ppc64 arch are:

        /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.4.ael7b.ppc64le
        /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.4.ael7b.ppc64le

        The version that I have tested with success is:
        /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.5.ael7b.ppc64le

        Please note the comments in 121669 :
        "It's good to keep in mind that even if they "fix" it, both JDK7 and JDK8 have releases out there that are reporting "ppc64" instead of "ppc64" for PPC64LE systems, so everyone will have to either:
        1. check for all conditions (os.arch "ppc64le" OR os.arch "ppc64" && sun.cpu.endian "little") for backward compatibility with JDK7/JDK8 JVMs (lets hope no other JVM decides to use "ppc64" without a way to tell the endianness)
        2. Ignore releases that report "ppc64" for PPC64LE (and tell users not to use those versions)
        Anyway, should they decide to fix it anyone depending on JDK9+ only (or ignoring backward compatibility on JDK7/8) will be happy enough.
        Hopefully, the life of 2.5.x on ppc64le platforms will be short-lived. 2.6.0 is due to be released, following the upstream release of 7u80 in April."

        It is not perfectly clear for me. At least, I understand that there will be a new 1.7 version in April, and that JDK8 also has this issue. Not sure if they plan to fix it. I do not use Java8.

        Show
        Tony Reix added a comment - The 2 versions that shown issue with ppc64 arch are: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.4.ael7b.ppc64le /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.4.ael7b.ppc64le The version that I have tested with success is: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.75-2.5.4.5.ael7b.ppc64le Please note the comments in 121669 : "It's good to keep in mind that even if they "fix" it, both JDK7 and JDK8 have releases out there that are reporting "ppc64" instead of "ppc64" for PPC64LE systems, so everyone will have to either: 1. check for all conditions (os.arch "ppc64le" OR os.arch "ppc64" && sun.cpu.endian "little") for backward compatibility with JDK7/JDK8 JVMs (lets hope no other JVM decides to use "ppc64" without a way to tell the endianness) 2. Ignore releases that report "ppc64" for PPC64LE (and tell users not to use those versions) Anyway, should they decide to fix it anyone depending on JDK9+ only (or ignoring backward compatibility on JDK7/8) will be happy enough. Hopefully, the life of 2.5.x on ppc64le platforms will be short-lived. 2.6.0 is due to be released, following the upstream release of 7u80 in April." It is not perfectly clear for me. At least, I understand that there will be a new 1.7 version in April, and that JDK8 also has this issue. Not sure if they plan to fix it. I do not use Java8.
        Hide
        Colin Patrick McCabe added a comment -

        Thanks, Tony. Can you add the information about which releases of OpenJDK have this problem to the description of this JIRA? That will help people who are trying to compile.

        Show
        Colin Patrick McCabe added a comment - Thanks, Tony. Can you add the information about which releases of OpenJDK have this problem to the description of this JIRA? That will help people who are trying to compile.
        Hide
        Tony Reix added a comment -

        OpenJDK team has decided to have arch="ppc64le" on PPC64LE architecture. And "mvn compile/package" with Hadoop 2.4.1 is now OK. Good !
        See: https://bugzilla.linux.ibm.com/show_bug.cgi?id=121669

        Show
        Tony Reix added a comment - OpenJDK team has decided to have arch="ppc64le" on PPC64LE architecture. And "mvn compile/package" with Hadoop 2.4.1 is now OK. Good ! See: https://bugzilla.linux.ibm.com/show_bug.cgi?id=121669
        Hide
        Colin Patrick McCabe added a comment -

        If this is concern, then why don't you write a test that cmake can execute to determine the endianness of the ARM machine directly? See how we handle HAVE_BETTER_TLS and HAVE_INTEL_SSE_INTRINSICS.

        Show
        Colin Patrick McCabe added a comment - If this is concern, then why don't you write a test that cmake can execute to determine the endianness of the ARM machine directly? See how we handle HAVE_BETTER_TLS and HAVE_INTEL_SSE_INTRINSICS .
        Hide
        Tony Reix added a comment -

        I do not understand the rationale for OpenJDK to consider PPC64LE machine as ppc64 architecture and ask all tools using a JVM to adapt their code in order to fix this strange choice. This change is not compatible with past.

        I'm using version 3.2.1 of Maven because other versions (older or fresher) have shown issues with Hadoop. This one seems stable.
        I do not understand why I should patch this version of Maven, or use a fresh new one because someone has decided that OpenJDK will mess up ppc64 and ppc64le.
        I do not understand why I should check that all the tools I'm using have been patched for adapting with the choice made by OpenJDK.
        I do not see the benefits of this choice. I only see drawbacks.

        Owner of https://bugzilla.linux.ibm.com/show_bug.cgi?id=121669 seems to want to "bring this up with upstream".

        Maven 3.2.1 and OpenJDK:
        $ mvn -version
        Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T11:37:52-06:00)
        Maven home: /opt/apache-maven-3.2.1
        Java version: 1.7.0_71, vendor: Oracle Corporation
        Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.4.ael7b.ppc64le/jre
        Default locale: en_US, platform encoding: ISO-8859-1
        OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64", family: "unix"

        Maven 3.2.1 and IBM JVM:
        $ mvn -version
        Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T11:37:52-06:00)
        Maven home: /opt/apache-maven-3.2.1
        Java version: 1.7.0, vendor: IBM Corporation
        Java home: /usr/lib/jvm/ibm-java-ppc64le-71/jre
        Default locale: en_US, platform encoding: ISO-8859-1
        OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64le", family: "unix"

        Show
        Tony Reix added a comment - I do not understand the rationale for OpenJDK to consider PPC64LE machine as ppc64 architecture and ask all tools using a JVM to adapt their code in order to fix this strange choice. This change is not compatible with past. I'm using version 3.2.1 of Maven because other versions (older or fresher) have shown issues with Hadoop. This one seems stable. I do not understand why I should patch this version of Maven, or use a fresh new one because someone has decided that OpenJDK will mess up ppc64 and ppc64le. I do not understand why I should check that all the tools I'm using have been patched for adapting with the choice made by OpenJDK. I do not see the benefits of this choice. I only see drawbacks. Owner of https://bugzilla.linux.ibm.com/show_bug.cgi?id=121669 seems to want to "bring this up with upstream". Maven 3.2.1 and OpenJDK: $ mvn -version Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T11:37:52-06:00) Maven home: /opt/apache-maven-3.2.1 Java version: 1.7.0_71, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.4.ael7b.ppc64le/jre Default locale: en_US, platform encoding: ISO-8859-1 OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64", family: "unix" Maven 3.2.1 and IBM JVM: $ mvn -version Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T11:37:52-06:00) Maven home: /opt/apache-maven-3.2.1 Java version: 1.7.0, vendor: IBM Corporation Java home: /usr/lib/jvm/ibm-java-ppc64le-71/jre Default locale: en_US, platform encoding: ISO-8859-1 OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64le", family: "unix"
        Hide
        Dinar Valeev added a comment -

        both BE and LE have ppc64 as javalibarch in openJDK since version 2.5.0. That was upstream descision to go that way.

        maven you're pointing to seems haven't catched up with that change yet.

        Note prior 2.5 we had ppc64le for LE.

        Show
        Dinar Valeev added a comment - both BE and LE have ppc64 as javalibarch in openJDK since version 2.5.0. That was upstream descision to go that way. maven you're pointing to seems haven't catched up with that change yet. Note prior 2.5 we had ppc64le for LE.
        Hide
        Tony Reix added a comment -

        Nuts ! The JIRA tools suppress blanks at beg of lines before the "^^^^^^^^^^^^".
        The examples of Maven are:

        WIth OpenJDK (RHEL 7.1 beta / PPC64LE), Maven says: arch=ppc64
        $ mvn -version
        ...
        OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64", family: "unix"
        .......................................................................................^^^^^^^^^^^

        WIth IBM JVM (RHEL 7.1 beta / PPC64LE), Maven says: arch=ppc64le
        $ mvn -version
        ...
        OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64le", family: "unix"
        .......................................................................................^^^^^^^^^^^^^^

        Show
        Tony Reix added a comment - Nuts ! The JIRA tools suppress blanks at beg of lines before the "^^^^^^^^^^^^". The examples of Maven are: WIth OpenJDK (RHEL 7.1 beta / PPC64LE), Maven says: arch=ppc64 $ mvn -version ... OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64", family: "unix" .......................................................................................^^^^^^^^^^^ WIth IBM JVM (RHEL 7.1 beta / PPC64LE), Maven says: arch=ppc64le $ mvn -version ... OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64le", family: "unix" .......................................................................................^^^^^^^^^^^^^^
        Hide
        Tony Reix added a comment -

        I think that this patch is wrong. The issue is not in Hadoop, but in OpenJDK, which says that arch=ppc64 (BE) instead of ppc64le.
        Instead of patching all tools that follow OpenJDK mistake on PPC64LE (saying it is ppc64 and thus Big-Endian), one has to patch OpenJDK only.

        I have opened (RHEL 7.1 beta / PPC64LE):
        https://bugzilla.linux.ibm.com/show_bug.cgi?id=121218
        https://bugzilla.linux.ibm.com/show_bug.cgi?id=121669
        Second one has shown issues with Maven.

        WIth OpenJDK (RHEL 7.1 beta / PPC64LE), Maven says:
        $ mvn -version
        Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T11:37:52-06:00)
        Maven home: /opt/apache-maven-3.2.1
        Java version: 1.7.0_71, vendor: Oracle Corporation
        Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.4.ael7b.ppc64le/jre
        Default locale: en_US, platform encoding: ISO-8859-1
        OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64", family: "unix"
        ^^^^^^^^^^^^^

        WIth IBM JVM (RHEL 7.1 beta / PPC64LE), Maven says (arch=ppc64le):
        $ mvn -version
        Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T15:58:10-05:00)
        Maven home: /opt/apache-maven-3.2.3
        Java version: 1.7.0, vendor: IBM Corporation
        Java home: /usr/lib/jvm/ibm-java-ppc64le-71/jre
        Default locale: en_US, platform encoding: UTF-8
        OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64le", family: "unix"
        ^^^^^^^^^^^^^

        Show
        Tony Reix added a comment - I think that this patch is wrong. The issue is not in Hadoop, but in OpenJDK, which says that arch=ppc64 (BE) instead of ppc64le. Instead of patching all tools that follow OpenJDK mistake on PPC64LE (saying it is ppc64 and thus Big-Endian), one has to patch OpenJDK only. I have opened (RHEL 7.1 beta / PPC64LE): https://bugzilla.linux.ibm.com/show_bug.cgi?id=121218 https://bugzilla.linux.ibm.com/show_bug.cgi?id=121669 Second one has shown issues with Maven. WIth OpenJDK (RHEL 7.1 beta / PPC64LE), Maven says: $ mvn -version Apache Maven 3.2.1 (ea8b2b07643dbb1b84b6d16e1f08391b666bc1e9; 2014-02-14T11:37:52-06:00) Maven home: /opt/apache-maven-3.2.1 Java version: 1.7.0_71, vendor: Oracle Corporation Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.71-2.5.3.4.ael7b.ppc64le/jre Default locale: en_US, platform encoding: ISO-8859-1 OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64", family: "unix" ^^^^^^^^^^^^^ WIth IBM JVM (RHEL 7.1 beta / PPC64LE), Maven says (arch=ppc64le): $ mvn -version Apache Maven 3.2.3 (33f8c3e1027c3ddde99d3cdebad2656a31e8fdf4; 2014-08-11T15:58:10-05:00) Maven home: /opt/apache-maven-3.2.3 Java version: 1.7.0, vendor: IBM Corporation Java home: /usr/lib/jvm/ibm-java-ppc64le-71/jre Default locale: en_US, platform encoding: UTF-8 OS name: "linux", version: "3.10.0-221.ael7b.ppc64le", arch: "ppc64le", family: "unix" ^^^^^^^^^^^^^
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #1843 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1843/)
        HADOOP-10968. hadoop native build fails to detect java_libarch on ppc64le (Dinar Valeev via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618919)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1843 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1843/ ) HADOOP-10968 . hadoop native build fails to detect java_libarch on ppc64le (Dinar Valeev via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618919 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Hide
        Hudson added a comment -

        SUCCESS: Integrated in Hadoop-Yarn-trunk #652 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/652/)
        HADOOP-10968. hadoop native build fails to detect java_libarch on ppc64le (Dinar Valeev via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618919)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #652 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/652/ ) HADOOP-10968 . hadoop native build fails to detect java_libarch on ppc64le (Dinar Valeev via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618919 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk #1868 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1868/)
        HADOOP-10968. hadoop native build fails to detect java_libarch on ppc64le (Dinar Valeev via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618919)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1868 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1868/ ) HADOOP-10968 . hadoop native build fails to detect java_libarch on ppc64le (Dinar Valeev via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618919 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Hide
        Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #6088 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6088/)
        HADOOP-10968. hadoop native build fails to detect java_libarch on ppc64le (Dinar Valeev via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618919)

        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
        • /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Show
        Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6088 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6088/ ) HADOOP-10968 . hadoop native build fails to detect java_libarch on ppc64le (Dinar Valeev via Colin Patrick McCabe) (cmccabe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1618919 ) /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/JNIFlags.cmake
        Hide
        Colin Patrick McCabe added a comment -

        P.S. Test failures are unrelated to this change to the native code

        Show
        Colin Patrick McCabe added a comment - P.S. Test failures are unrelated to this change to the native code
        Hide
        Colin Patrick McCabe added a comment -

        Thanks, Dinar. +1. Will commit to 2.6

        Show
        Colin Patrick McCabe added a comment - Thanks, Dinar. +1. Will commit to 2.6
        Hide
        Dinar Valeev added a comment -

        ppc64(BE) still should be fine,since libarch matches CMAKE_SYSTEM_PROCESSOR

        Show
        Dinar Valeev added a comment - ppc64(BE) still should be fine,since libarch matches CMAKE_SYSTEM_PROCESSOR
        Hide
        Colin Patrick McCabe added a comment -

        OK, so this patch only helps little-endian PPC machines. That's fine. Someone else can contribute a patch for the big-endian ones later if they want.

        I am +1 on this patch. Will wait a day or so to commit in case anyone else wants to comment.

        Show
        Colin Patrick McCabe added a comment - OK, so this patch only helps little-endian PPC machines. That's fine. Someone else can contribute a patch for the big-endian ones later if they want. I am +1 on this patch. Will wait a day or so to commit in case anyone else wants to comment.
        Hide
        Dinar Valeev added a comment -

        On ppc64 (big-endian) javaarch matches ppc64. While on ppc64le it can be ppc64(openjdk) or ppc64le(IBM Java).
        openJDK since 2.5 sets ppc64 as java arch, openJDK lower than 2.5 has ppc64le. IBM Java still sets ppc64le so we need that check for backward compatibility.
        It is no possible to run LE code on ppc64, so there is no chance to see ppc64le libarch on big-endian.

        Show
        Dinar Valeev added a comment - On ppc64 (big-endian) javaarch matches ppc64. While on ppc64le it can be ppc64(openjdk) or ppc64le(IBM Java). openJDK since 2.5 sets ppc64 as java arch, openJDK lower than 2.5 has ppc64le. IBM Java still sets ppc64le so we need that check for backward compatibility. It is no possible to run LE code on ppc64, so there is no chance to see ppc64le libarch on big-endian.
        Hide
        Colin Patrick McCabe added a comment -

        Thanks for this, Dinar.

        +    ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64le")
        +        IF(EXISTS "${_JAVA_HOME}/jre/lib/ppc64le")
        +                SET(_java_libarch "ppc64le")
        +        ELSE()
        +                SET(_java_libarch "ppc64")
        +        ENDIF()
        

        I don't understand how this works with big-endian powerpcs. I don't think this regex will match "powerpc" or "ppc", but only "powerpc64le" and "ppc64le", right? Do big-endian powerpcs show up as ppc64le? That seems odd.

        I also don't understand why we're checking for the existence of a directory to decide on the architecture... is this really the only option?

        Of course, I might be completely off base here, since I don't have access to any ppc systems.

        Show
        Colin Patrick McCabe added a comment - Thanks for this, Dinar. + ELSEIF (CMAKE_SYSTEM_PROCESSOR MATCHES "^(powerpc|ppc)64le" ) + IF(EXISTS "${_JAVA_HOME}/jre/lib/ppc64le" ) + SET(_java_libarch "ppc64le" ) + ELSE() + SET(_java_libarch "ppc64" ) + ENDIF() I don't understand how this works with big-endian powerpcs. I don't think this regex will match "powerpc" or "ppc", but only "powerpc64le" and "ppc64le", right? Do big-endian powerpcs show up as ppc64le? That seems odd. I also don't understand why we're checking for the existence of a directory to decide on the architecture... is this really the only option? Of course, I might be completely off base here, since I don't have access to any ppc systems.
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12661482/0001-Set-java_libarch-for-ppc64le-2.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        +1 javac. The applied patch does not increase the total number of javac compiler warnings.

        +1 javadoc. There were no new javadoc warning messages.

        +1 eclipse:eclipse. The patch built with eclipse:eclipse.

        +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

        +1 release audit. The applied patch does not increase the total number of release audit warnings.

        -1 core tests. The patch failed these unit tests in hadoop-common-project/hadoop-common:

        org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
        org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl
        org.apache.hadoop.ha.TestZKFailoverController
        org.apache.hadoop.ha.TestZKFailoverControllerStress
        org.apache.hadoop.ipc.TestDecayRpcScheduler

        +1 contrib tests. The patch passed contrib unit tests.

        Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4465//testReport/
        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4465//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12661482/0001-Set-java_libarch-for-ppc64le-2.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-common-project/hadoop-common: org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken org.apache.hadoop.metrics2.impl.TestMetricsSystemImpl org.apache.hadoop.ha.TestZKFailoverController org.apache.hadoop.ha.TestZKFailoverControllerStress org.apache.hadoop.ipc.TestDecayRpcScheduler +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4465//testReport/ Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4465//console This message is automatically generated.
        Hide
        Dinar Valeev added a comment -

        Fixed cmake syntax

        Show
        Dinar Valeev added a comment - Fixed cmake syntax
        Hide
        Hadoop QA added a comment -

        -1 overall. Here are the results of testing the latest attachment
        http://issues.apache.org/jira/secure/attachment/12661469/0001-Set-java_libarch-for-ppc64le-1.patch
        against trunk revision .

        +1 @author. The patch does not contain any @author tags.

        -1 tests included. The patch doesn't appear to include any new or modified tests.
        Please justify why no new tests are needed for this patch.
        Also please list what manual steps were performed to verify this patch.

        -1 javac. The patch appears to cause the build to fail.

        Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4463//console

        This message is automatically generated.

        Show
        Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12661469/0001-Set-java_libarch-for-ppc64le-1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4463//console This message is automatically generated.
        Hide
        Dinar Valeev added a comment -

        Improved version, check first if ppc64le is used.

        Note: IBM Java is using it.

        Show
        Dinar Valeev added a comment - Improved version, check first if ppc64le is used. Note: IBM Java is using it.
        Hide
        Dinar Valeev added a comment -

        This patch fixes an isssue

        Show
        Dinar Valeev added a comment - This patch fixes an isssue

          People

          • Assignee:
            Unassigned
            Reporter:
            Dinar Valeev
          • Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development