Details
-
Type:
Sub-task
-
Status: Resolved
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: None
-
Fix Version/s: 2.8.0, 3.0.0-alpha1
-
Component/s: None
-
Labels:None
-
Environment:
Solaris 11.2
Description
Compiling the Yarn Node Manager results in "fts" not found. On Solaris we have an alternative ftw with similar functionality.
This is isolated to a single file container-executor.c
Note that this will just fix the compilation error. A more serious issue is that Solaris does not support cgroups as Linux does.
Activity
- All
- Comments
- Work Log
- History
- Activity
- Transitions
I have already ported to the ftw library ( It works on Hadoop 2.2, both
Sparc and Intel)
I'll send you the code.
BTW, if we are working on YARN now, then there are some more native
issues which need to be filed, also in the HDFS module.
On another note, my boss asked whether we have a timeline for when we
can inform partners that Hadoop native runs on Solaris.
Which version of Hadoop are we targetting (my guess is 2.8), and when
can we expect to have the bits in the community.
To this end, I think we also need to decide what bugs are show stoppers,
and what are less critical, and how to prioritize them for release.
For example, on one extreme we could concentrate on releasing fixes that
enable a 32-bit version pretty soon, versus a 64-bit version that may
take longer. Also, targeting Solaris 11.2 versus Solaris 12 probably
requires a bit more work.
What do you think ?
if they are in HDFS please log them under the HDFS top-level JIRA that I opened rather than the YARN one, see HDFS-8478
There's little point in targeting the 32-bit JVM as, apart from the CMake adjustments, the code changes to build 64-bit on Solaris are mostly identical to those needed for 32-bit. Also, as of Java8 there is no 32-bit JVM for Solaris, only 64-bit is available - http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8023288 and Java7 is EOL - https://www.java.com/en/download/faq/java_7.xml
I think it's premature at this point to talk about timelines as we are still evaluating the work that needs to be done and any targets we gave would just be guesses.
15778275 is the Solaris bug that has been logged to investigate adding the fts interfaces to Solaris
See also the discussion in http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201506.mbox/%3C558BCA3A.1020602%40oracle.com%3E. The use of fts(3) should be replaced by nftw(3)
 -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | patch | 0m 1s | The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. |
| 0 | pre-patch | 5m 21s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 7m 40s | There were no new javac warning messages. |
| +1 | release audit | 0m 19s | The applied patch does not increase the total number of release audit warnings. |
| -1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | install | 1m 20s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
| +1 | yarn tests | 6m 6s | Tests passed in hadoop-yarn-server-nodemanager. |
| 21m 21s |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12746619/YARN-3724 |
| Optional Tests | javac unit |
| git revision | trunk / 06e5dd2 |
| whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8621/artifact/patchprocess/whitespace.txt |
| hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8621/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8621/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8621/console |
This message was automatically generated.
| -1 overall |
| Vote | Subsystem | Runtime | Comment |
|---|---|---|---|
| 0 | pre-patch | 5m 20s | Pre-patch trunk compilation is healthy. |
| +1 | @author | 0m 0s | The patch does not contain any @author tags. |
| -1 | tests included | 0m 0s | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. |
| +1 | javac | 7m 42s | There were no new javac warning messages. |
| +1 | release audit | 0m 19s | The applied patch does not increase the total number of release audit warnings. |
| -1 | whitespace | 0m 0s | The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. |
| +1 | install | 1m 20s | mvn install still works. |
| +1 | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. |
| -1 | yarn tests | 6m 2s | Tests failed in hadoop-yarn-server-nodemanager. |
| 21m 19s |
| Reason | Tests |
|---|---|
| Failed unit tests | hadoop.yarn.server.nodemanager.TestDeletionService |
| Subsystem | Report/Notes |
|---|---|
| Patch URL | http://issues.apache.org/jira/secure/attachment/12746793/YARN-3724.001.patch |
| Optional Tests | javac unit |
| git revision | trunk / ee98d63 |
| whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8634/artifact/patchprocess/whitespace.txt |
| hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8634/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt |
| Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8634/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8634/console |
This message was automatically generated.
Alan Burlison, please use the "Target Version" field to express your intention. Fix-version is exclusively used by committers when a patch gets committed.
IAC, 2.7.1 is already done. Targeting this for 2.7.2/2.8.0.
+1 committed to trunk and branch-2.
FAILURE: Integrated in Hadoop-trunk-Commit #8704 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8704/)
YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
FAILURE: Integrated in Hadoop-Yarn-trunk #1317 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1317/)
YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #594 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/594/)
YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #582 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/582/)
YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)
- hadoop-yarn-project/CHANGES.txt
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
FAILURE: Integrated in Hadoop-Hdfs-trunk #2473 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2473/)
YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)
- hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
- hadoop-yarn-project/CHANGES.txt
I'm working on a replacement for the Linux FTS code using POSIX FTW APIs instead. As you say the Cgroups issue is a more tricky one, and I feel deserves it's own bug. I've therefore logged:
YARN-3718 hadoop-yarn-server-nodemanager's use of Linux Cgroups is non-portable