Thanks for taking this up Kirill. I verified that the native library builds on OSX with your changes.

Few comments from the 2.0.5 patch.

1. Tests are not able to load the native library. TestNativeIO fails with "Unable to load native-hadoop library for your platform... using builtin-java classes where applicable".
2. The working directory should be restored before returning to avoid changing the callers cwd. I think it would be best to use the mkdirat and openat functions except on OS X via conditional compilation.

   -    if (mkdirat(cwd, token, perm) != 0) {
   +    if(fchdir(cwd) < 0) {
   

3. The iterative approach to replace fcloseall looks somewhat inefficient. At the very least it should be restricted to OS X and fcloseall() used everywhere else.

   +  for (; rl.rlim_max > 2; rl.rlim_max--) {
   +    close(rl.rlim_max);
   +  }
   

4. Is this change necessary? I can build without it.

   -        <version>2.4.0a</version>
   +        <version>2.5.0</version>
   

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12588023/HADOOP-9648-native-osx.2.0.5-alpha-rc1.patch
against trunk revision .

-1 patch. The patch command could not apply the patch.

Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3340//console

This message is automatically generated.

Fix some issues of the original patch and fix related test failures in test-container-executor, changes:
1. Issue: setnetgrent is deleted in original code, this is not right, we don't need if test but still need to call setnetgrent
2. Issue: mkdirs skip create dir if the path exists, but if the path is a file, it can still succeed. Fix: change the whole implementation, mkdirat, openat is not needed anymore
3. LOGIN_NAME_MAX is not present in macos, changed to use sysconf
4. fcloseall is not present in macos, changed to close opened fds(stdin, stdout, stderr)
5. macos/freebsd do not have cgroup, disable and print error message
test-container-executer issues:
6. macosx do not have user bin, skip a test
7. macosx /etc/passwd is not real path(have symlink), changed to /bin/ls

Now compile with native and test-container-executor can run successfully in my macbook.

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. The javadoc tool did not generate any warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3341//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3341//console

This message is automatically generated.

LGTM, I could compile native library with the v2 patch on my MacBook (OS X 10.9).

The common changes look good to me. I can't really evaluate the YARN changes though, and I don't actually have a Mac to test this. Maybe a Mac-capable developer can step in and do the final review and +1?

This patch works for me, but I've only tested the client. Not sure if anyone actually plans to run YARN containers on a cluster of Macs (or even if that's possible to do, even with this patch) so the hadoop-yarn-project changes are harder to test in a real environment ...

If committers are not fully comfortable with making the proposed changes to hadoop-yarn-server, then maybe this could be broken up into 2 issues?

1) The hadoop-common-project changes. Fixing that would allow libhadoop.dylib to build on Mac OS X and fix the client (I duplicated this part of the issue unknowingly in HADOOP-10358).

2) The hadoop-yarn-project changes. This might be seen as the more risky part of this patch, and maybe less useful since I suspect most people interested in getting native code working on Macs run hadoop on Linux clusters but interact with them from Mac clients (I could be wrong of course).

I just spent a while tracking this down and fixing it. Then I came here to open a JIRA and submit a patchset, then I found there already was one open. It would be great to have this merged.

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

-1 core tests. The following test timeouts occurred in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager:

org.apache.hadoop.http.TestHttpServerLifecycle

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/3822//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/3822//console

This message is automatically generated.

+1. It would be good to get this committed. This will help enable a lot of desktop/laptop testing of things that require native libraries.

Hi Vinod Kumar Vavilapalli or Jason Lowe, I see the the latest container-executor changes are done by you, currently the main concern of this jira is yarn related native code changes, you seems the right person to ask for help, could you give some comments about this?

This is primarily a YARN change rather than a change to common. We either should have this JIRA track the hadoop-common change and file a separate for the container-executor fixes or move this to YARN and commit there.

Otherwise patch looks good overall, mostly nits:

• no check for strdup returning NULL
• should have braces even for 1-liners, e.g.: while (*p == '/')
• "Failed to mount cgroup controller, not support" should be "Failed to mount cgroup controller, not supported"

I moved this jira to YARN, and update the path, unfortunately I found more buggy native code on in hdfs, so I think it's better to separate those code into three jiras.

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3988//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3988//console

This message is automatically generated.

+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12617363/HADOOP-9648.v2.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

+1 tests included. The patch appears to include 1 new or modified test files.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4067//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4067//console

This message is automatically generated.

Created HDFS-6534 for HDFS parts, and YARN-2161 for YARN parts, this jira is changed to on cover Common parts

-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12650515/HADOOP-10699-common.v3.patch
against trunk revision .

+1 @author. The patch does not contain any @author tags.

-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.

+1 javac. The applied patch does not increase the total number of javac compiler warnings.

+1 javadoc. There were no new javadoc warning messages.

+1 eclipse:eclipse. The patch built with eclipse:eclipse.

+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

+1 release audit. The applied patch does not increase the total number of release audit warnings.

+1 core tests. The patch passed unit tests in hadoop-common-project/hadoop-common.

+1 contrib tests. The patch passed contrib unit tests.

Test results: https://builds.apache.org/job/PreCommit-HADOOP-Build/4073//testReport/
Console output: https://builds.apache.org/job/PreCommit-HADOOP-Build/4073//console

This message is automatically generated.

+1 lgtm, committing this.

Thanks, Binglin! I committed this to trunk and branch-2.

SUCCESS: Integrated in Hadoop-trunk-Commit #5717 (See https://builds.apache.org/job/Hadoop-trunk-Commit/5717/)
HADOOP-10699. Fix build native library on mac osx. Contributed by Binglin Chang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603042)

• /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
• /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c

SUCCESS: Integrated in Hadoop-Yarn-trunk #586 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/586/)
HADOOP-10699. Fix build native library on mac osx. Contributed by Binglin Chang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603042)

• /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
• /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c

SUCCESS: Integrated in Hadoop-Hdfs-trunk #1777 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1777/)
HADOOP-10699. Fix build native library on mac osx. Contributed by Binglin Chang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603042)

• /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
• /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c

FAILURE: Integrated in Hadoop-Mapreduce-trunk #1804 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1804/)
HADOOP-10699. Fix build native library on mac osx. Contributed by Binglin Chang (jlowe: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603042)

• /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
• /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/native/src/org/apache/hadoop/security/JniBasedUnixGroupsNetgroupMapping.c