Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None

      Description

      When compiling on macosx with -Pnative, there are several warning and errors, fix this would help hadoop developers with macosx env.

      1. YARN-2161.v2.patch
        9 kB
        Binglin Chang
      2. YARN-2161.v1.patch
        7 kB
        Binglin Chang

        Issue Links

          Activity

          Hide
          decster Binglin Chang added a comment -

          Changes:

          container-executor.c:
          1. make mkdirs more compatible, remove usage of mkdirat/openat
          2. use sysconf() to get LOGIN_NAME_MAX
          3. macosx doesn't have fcloseall, so close all opened fds on macosx
          4. disable cgroup on macosx

          test-container-executor.c:
          1. macosx do not have user bin, skip a check
          2. change /etc/passwd(not exists on mac) to /bin/ls

          Show
          decster Binglin Chang added a comment - Changes: container-executor.c: 1. make mkdirs more compatible, remove usage of mkdirat/openat 2. use sysconf() to get LOGIN_NAME_MAX 3. macosx doesn't have fcloseall, so close all opened fds on macosx 4. disable cgroup on macosx test-container-executor.c: 1. macosx do not have user bin, skip a check 2. change /etc/passwd(not exists on mac) to /bin/ls
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12650514/YARN-2161.v1.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3991//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3991//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12650514/YARN-2161.v1.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3991//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3991//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          Looks good for the most part! A few minor things:

          +#if defined(__MACH__)
          +  // only those fds are opened assuming no bug
          +  fclose(LOGFILE);
          +  fclose(ERRORFILE);
          +  fclose(stdin);
          +  fclose(stdout);
          +  fclose(stderr);
          +#else
             fcloseall();
          +#endif
          

          It would probably be better to detect if fcloseall is a valid function. If not, then use the manual fclose. This would allow the code to run on not just OS X but other systems as well.

          +#if defined(__FreeBSD__) || defined(__MACH__)
          +  fprintf(LOGFILE, "Failed to mount cgroup controller, not supported\n");
          +  return -1;
          +#else
          

          I sort of prefer the fix from YARN-1327 here, where it does +#ifndef __linux instead.

          +#if !defined(__MACH__) // macosx do not have user bin
             if (check_user("bin") == NULL) {
               printf("FAIL: failed check for whitelisted system user bin\n");
               exit(1);
             }
          +#endif
          

          Rather than ignore this test completely, can we chose a different user instead?

          Also, can you incorporate the other changes from YARN-1327 so that we can do one commit?

          Thanks!

          Show
          aw Allen Wittenauer added a comment - Looks good for the most part! A few minor things: +# if defined(__MACH__) + // only those fds are opened assuming no bug + fclose(LOGFILE); + fclose(ERRORFILE); + fclose(stdin); + fclose(stdout); + fclose(stderr); +# else fcloseall(); +#endif It would probably be better to detect if fcloseall is a valid function. If not, then use the manual fclose. This would allow the code to run on not just OS X but other systems as well. +# if defined(__FreeBSD__) || defined(__MACH__) + fprintf(LOGFILE, "Failed to mount cgroup controller, not supported\n" ); + return -1; +# else I sort of prefer the fix from YARN-1327 here, where it does +#ifndef __linux instead. +# if !defined(__MACH__) // macosx do not have user bin if (check_user( "bin" ) == NULL) { printf( "FAIL: failed check for whitelisted system user bin\n" ); exit(1); } +#endif Rather than ignore this test completely, can we chose a different user instead? Also, can you incorporate the other changes from YARN-1327 so that we can do one commit? Thanks!
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12650514/YARN-2161.v1.patch
          against trunk revision b93d960.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5089//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5089//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12650514/YARN-2161.v1.patch against trunk revision b93d960. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5089//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5089//console This message is automatically generated.
          Hide
          decster Binglin Chang added a comment -

          Thanks for the review Allen. Attach new version of the patch, changes:
          1. use cmake CHECK_FUNCTION_EXISTS to check fcloseall exists
          2. change user bin to user daemon (which both linux and macosx have)
          3. add some fix in YARN-1327 (setpgid and libgen.h)

          Show
          decster Binglin Chang added a comment - Thanks for the review Allen. Attach new version of the patch, changes: 1. use cmake CHECK_FUNCTION_EXISTS to check fcloseall exists 2. change user bin to user daemon (which both linux and macosx have) 3. add some fix in YARN-1327 (setpgid and libgen.h)
          Hide
          hadoopqa Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12670922/YARN-2161.v2.patch
          against trunk revision ef784a2.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5094//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5094//console

          This message is automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12670922/YARN-2161.v2.patch against trunk revision ef784a2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5094//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5094//console This message is automatically generated.
          Hide
          aw Allen Wittenauer added a comment -

          +1 lgtm. Committing to branch-2 and trunk.

          Thanks!

          Show
          aw Allen Wittenauer added a comment - +1 lgtm. Committing to branch-2 and trunk. Thanks!
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk #691 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/691/)
          YARN-2161. Fix build on macosx: YARN parts (Binglin Chang via aw) (aw: rev 034df0e2eb2824fb46a1e75b52d43d9914a04e56)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/config.h.cmake
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #691 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/691/ ) YARN-2161 . Fix build on macosx: YARN parts (Binglin Chang via aw) (aw: rev 034df0e2eb2824fb46a1e75b52d43d9914a04e56) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/config.h.cmake hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Hdfs-trunk #1882 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1882/)
          YARN-2161. Fix build on macosx: YARN parts (Binglin Chang via aw) (aw: rev 034df0e2eb2824fb46a1e75b52d43d9914a04e56)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/config.h.cmake
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          • hadoop-yarn-project/CHANGES.txt
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Hadoop-Hdfs-trunk #1882 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1882/ ) YARN-2161 . Fix build on macosx: YARN parts (Binglin Chang via aw) (aw: rev 034df0e2eb2824fb46a1e75b52d43d9914a04e56) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/config.h.cmake hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c hadoop-yarn-project/CHANGES.txt
          Hide
          hudson Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1907 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1907/)
          YARN-2161. Fix build on macosx: YARN parts (Binglin Chang via aw) (aw: rev 034df0e2eb2824fb46a1e75b52d43d9914a04e56)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/config.h.cmake
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
          Show
          hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1907 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1907/ ) YARN-2161 . Fix build on macosx: YARN parts (Binglin Chang via aw) (aw: rev 034df0e2eb2824fb46a1e75b52d43d9914a04e56) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/config.h.cmake hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
          Hide
          xgong Xuan Gong added a comment -

          Binglin Chang Allen Wittenauer
          For fixing YARN-2701, i need to revert the native code changes for mkdirs in container-executor.c

          Show
          xgong Xuan Gong added a comment - Binglin Chang Allen Wittenauer For fixing YARN-2701 , i need to revert the native code changes for mkdirs in container-executor.c
          Hide
          xgong Xuan Gong added a comment -

          The changes for mkdirs in container-executor.c bring the race condition when two containers are trying to check and create directory at the same time.

          Show
          xgong Xuan Gong added a comment - The changes for mkdirs in container-executor.c bring the race condition when two containers are trying to check and create directory at the same time.
          Hide
          decster Binglin Chang added a comment -

          Hi Xuan Gong, sorry for break the code. I see in YARN-2701 you already have fix code, but decide to revert the code in the end to be more safe, but this breaks the mac build, how about use #ifdef to use old code when compiling on glibc > 2.10(http://linux.die.net/man/2/openat) and use your fixing code otherwise?

          Show
          decster Binglin Chang added a comment - Hi Xuan Gong , sorry for break the code. I see in YARN-2701 you already have fix code, but decide to revert the code in the end to be more safe, but this breaks the mac build, how about use #ifdef to use old code when compiling on glibc > 2.10( http://linux.die.net/man/2/openat ) and use your fixing code otherwise?
          Hide
          aw Allen Wittenauer added a comment -

          I've re-opened YARN-2701.

          Show
          aw Allen Wittenauer added a comment - I've re-opened YARN-2701 .

            People

            • Assignee:
              decster Binglin Chang
              Reporter:
              decster Binglin Chang
            • Votes:
              1 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development