Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6141

ppc64le on Linux doesn't trigger __linux get_executable codepath

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 3.0.0-alpha4
    • Fix Version/s: 2.9.0, 3.0.0-alpha4, 2.8.2
    • Component/s: nodemanager
    • Labels:
    • Environment:

      $ uname -a
      Linux f8eef0f055cf 3.16.0-30-generic #40~14.04.1-Ubuntu SMP Thu Jan 15 17:42:36 UTC 2015 ppc64le ppc64le ppc64le GNU/Linux

    • Hadoop Flags:
      Reviewed

      Description

      On ppc64le architecture, the build fails in the 'Hadoop YARN NodeManager' project with the below error:

      Cannot safely determine executable path with a relative HADOOP_CONF_DIR on this operating system.
      [WARNING] #error Cannot safely determine executable path with a relative HADOOP_CONF_DIR on this operating system.
      [WARNING] ^
      [WARNING] make[2]: *** [CMakeFiles/container.dir/main/native/container-executor/impl/get_executable.c.o] Error 1
      [WARNING] make[2]: *** Waiting for unfinished jobs....
      [WARNING] make[1]: *** [CMakeFiles/container.dir/all] Error 2
      [WARNING] make: *** [all] Error 2
      [INFO] ------------------------------------------------------------------------
      [INFO] BUILD FAILURE
      [INFO] ------------------------------------------------------------------------

      Cmake version used :
      $ /usr/bin/cmake --version
      cmake version 2.8.12.2

      1. YARN-6141.patch
        0.8 kB
        Sonia Garudi

        Activity

        Hide
        aw Allen Wittenauer added a comment -

        This is working as expected; this isn't a cmake problem. This is either a compiler/header problem or a build-time configuration problem.

        Specifically, on Linux platforms:

        #elif defined(__linux)
        
        
        char* get_executable(char *argv0) {
          return __get_exec_readproc("/proc/self/exe");
        }
        

        should be getting triggered. If it's not, then __linux isn't being defined. If ppc64le on Linux is not defining __linux, then I'd say the compiler chain is broken. If that is actually intentional, then supplemental code will need to be provided. If it can't be provided, then container-executor.conf.dir needs to be defined on the maven command line to point to an absolute path where the container-executor.conf will be located post-installation.

        Show
        aw Allen Wittenauer added a comment - This is working as expected; this isn't a cmake problem. This is either a compiler/header problem or a build-time configuration problem. Specifically, on Linux platforms: #elif defined(__linux) char * get_executable( char *argv0) { return __get_exec_readproc( "/proc/self/exe" ); } should be getting triggered. If it's not, then __linux isn't being defined. If ppc64le on Linux is not defining __linux, then I'd say the compiler chain is broken. If that is actually intentional, then supplemental code will need to be provided. If it can't be provided, then container-executor.conf.dir needs to be defined on the maven command line to point to an absolute path where the container-executor.conf will be located post-installation.
        Hide
        Sonia Sonia Garudi added a comment -

        Thank you for the response. On checking, I found that __linux macro is not defined for C99 on ppc64le.

        $ gcc -std=c99 -dM -E -xc /dev/null | grep linux
        #define _linux_ 1
        #define _gnu_linux_ 1

        In the src/main/native/container-executor/impl/get_executable.c file, I changed the macro from _linux to __linux_ (since this macro is predefined on ppc64le for C99).
        I got a successful build with this change . Attached a patch file with the same.

        Show
        Sonia Sonia Garudi added a comment - Thank you for the response. On checking, I found that __linux macro is not defined for C99 on ppc64le. $ gcc -std=c99 -dM -E -xc /dev/null | grep linux #define _ linux _ 1 #define _ gnu_linux _ 1 In the src/main/native/container-executor/impl/get_executable.c file, I changed the macro from _ linux to __linux _ (since this macro is predefined on ppc64le for C99). I got a successful build with this change . Attached a patch file with the same.
        Hide
        Sonia Sonia Garudi added a comment -

        On upgrading the version of cmake to >3.1 the build is successful. For cmake version < 3.1, the C standard is being set to c99 in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt:

        if (CMAKE_VERSION VERSION_LESS "3.1")
        		  # subset of CMAKE_<LANG>_COMPILER_ID
        		  # https://cmake.org/cmake/help/v3.0/variable/CMAKE_LANG_COMPILER_ID.html
        		  if (CMAKE_C_COMPILER_ID STREQUAL "GNU" OR
        			  CMAKE_C_COMPILER_ID STREQUAL "Clang" OR
        			  CMAKE_C_COMPILER_ID STREQUAL "AppleClang")
        			set (CMAKE_C_FLAGS "-std=c99 -Wall -pedantic-errors ${CMAKE_C_FLAGS}")
        		  elseif (CMAKE_C_COMPILER_ID STREQUAL "Intel")
        			set (CMAKE_C_FLAGS "-std=c99 -Wall ${CMAKE_C_FLAGS}")
        		  elseif (CMAKE_C_COMPILER_ID STREQUAL "SunPro")
        			set (CMAKE_C_FLAGS "-xc99 ${CMAKE_C_FLAGS}")
        		  endif ()
        		else ()
        		  set (CMAKE_C_STANDARD 99)
        		endif ()
        
        Show
        Sonia Sonia Garudi added a comment - On upgrading the version of cmake to >3.1 the build is successful. For cmake version < 3.1, the C standard is being set to c99 in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt: if (CMAKE_VERSION VERSION_LESS "3.1" ) # subset of CMAKE_<LANG>_COMPILER_ID # https: //cmake.org/cmake/help/v3.0/variable/CMAKE_LANG_COMPILER_ID.html if (CMAKE_C_COMPILER_ID STREQUAL "GNU" OR CMAKE_C_COMPILER_ID STREQUAL "Clang" OR CMAKE_C_COMPILER_ID STREQUAL "AppleClang" ) set (CMAKE_C_FLAGS "-std=c99 -Wall -pedantic-errors ${CMAKE_C_FLAGS}" ) elseif (CMAKE_C_COMPILER_ID STREQUAL "Intel" ) set (CMAKE_C_FLAGS "-std=c99 -Wall ${CMAKE_C_FLAGS}" ) elseif (CMAKE_C_COMPILER_ID STREQUAL "SunPro" ) set (CMAKE_C_FLAGS "-xc99 ${CMAKE_C_FLAGS}" ) endif () else () set (CMAKE_C_STANDARD 99) endif ()
        Hide
        Ayappan Ayappan added a comment - - edited

        The predefined macros "_linux" and "linux" are not POSIX-compliant (neither are unix and __unix ), so ppc64le is in the right here. They are allowed with GNU extensions, but should not be allowed for strict ANSI, which is implied by -std=c99. The Hadoop code should be changed to use the compliant __linux_ .

        See https://gcc.gnu.org/onlinedocs/cpp/System-specific-Predefined-Macros.html#System-specific-Predefined-Macros for discussion and guidance. In particular:

        "We are slowly phasing out all predefined macros which are outside the reserved namespace. You should never use them in new programs, and we encourage you to correct older code to use the parallel macros whenever you find it." Here the "parallel macros" are those with underscores both before and after the name, such as _linux_ .

        I do not know why the POWER back end is the only one that enforces this, but that point is moot. Hadoop code should be changed.

        Show
        Ayappan Ayappan added a comment - - edited The predefined macros "_ linux" and "linux" are not POSIX-compliant (neither are unix and __unix ), so ppc64le is in the right here. They are allowed with GNU extensions, but should not be allowed for strict ANSI, which is implied by -std=c99. The Hadoop code should be changed to use the compliant __linux _ . See https://gcc.gnu.org/onlinedocs/cpp/System-specific-Predefined-Macros.html#System-specific-Predefined-Macros for discussion and guidance. In particular: "We are slowly phasing out all predefined macros which are outside the reserved namespace. You should never use them in new programs, and we encourage you to correct older code to use the parallel macros whenever you find it." Here the "parallel macros" are those with underscores both before and after the name, such as _ linux _ . I do not know why the POWER back end is the only one that enforces this, but that point is moot. Hadoop code should be changed.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 mvninstall 13m 30s trunk passed
        +1 compile 0m 30s trunk passed
        +1 mvnsite 0m 29s trunk passed
        +1 mvneclipse 0m 15s trunk passed
        +1 mvninstall 0m 25s the patch passed
        +1 compile 0m 27s the patch passed
        +1 cc 0m 27s the patch passed
        +1 javac 0m 27s the patch passed
        +1 mvnsite 0m 26s the patch passed
        +1 mvneclipse 0m 12s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 unit 13m 20s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 17s The patch does not generate ASF License warnings.
        30m 41s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-6141
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859082/YARN-6141.patch
        Optional Tests asflicense compile cc mvnsite javac unit
        uname Linux 8dbe6d22e5f2 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 6d95866
        Default Java 1.8.0_121
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15298/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/15298/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 13m 30s trunk passed +1 compile 0m 30s trunk passed +1 mvnsite 0m 29s trunk passed +1 mvneclipse 0m 15s trunk passed +1 mvninstall 0m 25s the patch passed +1 compile 0m 27s the patch passed +1 cc 0m 27s the patch passed +1 javac 0m 27s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 12s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 unit 13m 20s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 30m 41s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-6141 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12859082/YARN-6141.patch Optional Tests asflicense compile cc mvnsite javac unit uname Linux 8dbe6d22e5f2 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 6d95866 Default Java 1.8.0_121 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/15298/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/15298/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        Ayappan Ayappan added a comment -

        Any update on this ?

        Show
        Ayappan Ayappan added a comment - Any update on this ?
        Hide
        Sonia Sonia Garudi added a comment -

        Any update on this ?

        Show
        Sonia Sonia Garudi added a comment - Any update on this ?
        Hide
        Sonia Sonia Garudi added a comment -

        Any update ?

        Show
        Sonia Sonia Garudi added a comment - Any update ?
        Hide
        Sonia Sonia Garudi added a comment -

        This was a build failure due to undefined macro on ppc64le platform. The patch resolves this issue and the build is successul.

        Show
        Sonia Sonia Garudi added a comment - This was a build failure due to undefined macro on ppc64le platform. The patch resolves this issue and the build is successul.
        Hide
        Ayappan Ayappan added a comment -

        Allen Wittenauer Any update on this ?

        Show
        Ayappan Ayappan added a comment - Allen Wittenauer Any update on this ?
        Hide
        Ayappan Ayappan added a comment -

        Any update here guys ?

        Show
        Ayappan Ayappan added a comment - Any update here guys ?
        Hide
        Ayappan Ayappan added a comment -

        What is the inhibitor here?

        As mentioned earlier, the _linux_ is recommended by gcc community. https://gcc.gnu.org/onlinedocs/cpp/System-specific-Predefined-Macros.html#System-specific-Predefined-Macros
        Going forward this is going to be the future standard.
        This patch won't break any platform but just complying to the standards.

        Show
        Ayappan Ayappan added a comment - What is the inhibitor here? As mentioned earlier, the _ linux _ is recommended by gcc community. https://gcc.gnu.org/onlinedocs/cpp/System-specific-Predefined-Macros.html#System-specific-Predefined-Macros Going forward this is going to be the future standard. This patch won't break any platform but just complying to the standards.
        Hide
        Ayappan Ayappan added a comment -

        Any update here ?
        Thanks

        Show
        Ayappan Ayappan added a comment - Any update here ? Thanks
        Hide
        Sonia Sonia Garudi added a comment -

        Any update on this ?

        Show
        Sonia Sonia Garudi added a comment - Any update on this ?
        Hide
        Ayappan Ayappan added a comment -

        Hi Guys,
        Any update on this ?

        Show
        Ayappan Ayappan added a comment - Hi Guys, Any update on this ?
        Hide
        ajisakaa Akira Ajisaka added a comment -

        LGTM, +1. Checking this in.

        Show
        ajisakaa Akira Ajisaka added a comment - LGTM, +1. Checking this in.
        Hide
        ajisakaa Akira Ajisaka added a comment -

        Committed this to trunk, branch-2, branch-2.8, and branch-2.8.1. Thanks Ayappan and Sonia Garudi for the contribution.

        Show
        ajisakaa Akira Ajisaka added a comment - Committed this to trunk, branch-2, branch-2.8, and branch-2.8.1. Thanks Ayappan and Sonia Garudi for the contribution.
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11779/)
        YARN-6141. ppc64le on Linux doesn't trigger __linux get_executable (aajisaka: rev bc28da65fb1c67904aa3cefd7273cb7423521014)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.c
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11779 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11779/ ) YARN-6141 . ppc64le on Linux doesn't trigger __linux get_executable (aajisaka: rev bc28da65fb1c67904aa3cefd7273cb7423521014) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.c
        Hide
        Ayappan Ayappan added a comment -
        Show
        Ayappan Ayappan added a comment - Thanks Akira Ajisaka
        Hide
        Sonia Sonia Garudi added a comment -

        Thanks Akira Ajisaka .

        Show
        Sonia Sonia Garudi added a comment - Thanks Akira Ajisaka .
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - 2.8.1 became a security release. Moving fix-version to 2.8.2 after the fact.

          People

          • Assignee:
            Ayappan Ayappan
            Reporter:
            Sonia Sonia Garudi
          • Votes:
            1 Vote for this issue
            Watchers:
            7 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development