Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Environment:

      Solaris 11.2

    • Target Version/s:
    • External issue ID:
      15778275 : SUNBT7152812 libc should provide fts interfaces for traversing file hierarchy

      Description

      Compiling the Yarn Node Manager results in "fts" not found. On Solaris we have an alternative ftw with similar functionality.
      This is isolated to a single file container-executor.c
      Note that this will just fix the compilation error. A more serious issue is that Solaris does not support cgroups as Linux does.

        Activity

        Hide
        alanburlison Alan Burlison added a comment -

        I'm working on a replacement for the Linux FTS code using POSIX FTW APIs instead. As you say the Cgroups issue is a more tricky one, and I feel deserves it's own bug. I've therefore logged:

        YARN-3718 hadoop-yarn-server-nodemanager's use of Linux Cgroups is non-portable

        Show
        alanburlison Alan Burlison added a comment - I'm working on a replacement for the Linux FTS code using POSIX FTW APIs instead. As you say the Cgroups issue is a more tricky one, and I feel deserves it's own bug. I've therefore logged: YARN-3718 hadoop-yarn-server-nodemanager's use of Linux Cgroups is non-portable
        Hide
        malcolm kavalsky Malcolm Kavalsky added a comment -

        I have already ported to the ftw library ( It works on Hadoop 2.2, both
        Sparc and Intel)

        I'll send you the code.

        Show
        malcolm kavalsky Malcolm Kavalsky added a comment - I have already ported to the ftw library ( It works on Hadoop 2.2, both Sparc and Intel) I'll send you the code.
        Hide
        malcolm kavalsky Malcolm Kavalsky added a comment -

        BTW, if we are working on YARN now, then there are some more native
        issues which need to be filed, also in the HDFS module.

        On another note, my boss asked whether we have a timeline for when we
        can inform partners that Hadoop native runs on Solaris.
        Which version of Hadoop are we targetting (my guess is 2.8), and when
        can we expect to have the bits in the community.
        To this end, I think we also need to decide what bugs are show stoppers,
        and what are less critical, and how to prioritize them for release.

        For example, on one extreme we could concentrate on releasing fixes that
        enable a 32-bit version pretty soon, versus a 64-bit version that may
        take longer. Also, targeting Solaris 11.2 versus Solaris 12 probably
        requires a bit more work.

        What do you think ?

        Show
        malcolm kavalsky Malcolm Kavalsky added a comment - BTW, if we are working on YARN now, then there are some more native issues which need to be filed, also in the HDFS module. On another note, my boss asked whether we have a timeline for when we can inform partners that Hadoop native runs on Solaris. Which version of Hadoop are we targetting (my guess is 2.8), and when can we expect to have the bits in the community. To this end, I think we also need to decide what bugs are show stoppers, and what are less critical, and how to prioritize them for release. For example, on one extreme we could concentrate on releasing fixes that enable a 32-bit version pretty soon, versus a 64-bit version that may take longer. Also, targeting Solaris 11.2 versus Solaris 12 probably requires a bit more work. What do you think ?
        Hide
        alanburlison Alan Burlison added a comment -

        if they are in HDFS please log them under the HDFS top-level JIRA that I opened rather than the YARN one, see HDFS-8478

        There's little point in targeting the 32-bit JVM as, apart from the CMake adjustments, the code changes to build 64-bit on Solaris are mostly identical to those needed for 32-bit. Also, as of Java8 there is no 32-bit JVM for Solaris, only 64-bit is available - http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8023288 and Java7 is EOL - https://www.java.com/en/download/faq/java_7.xml

        I think it's premature at this point to talk about timelines as we are still evaluating the work that needs to be done and any targets we gave would just be guesses.

        Show
        alanburlison Alan Burlison added a comment - if they are in HDFS please log them under the HDFS top-level JIRA that I opened rather than the YARN one, see HDFS-8478 There's little point in targeting the 32-bit JVM as, apart from the CMake adjustments, the code changes to build 64-bit on Solaris are mostly identical to those needed for 32-bit. Also, as of Java8 there is no 32-bit JVM for Solaris, only 64-bit is available - http://bugs.java.com/bugdatabase/view_bug.do?bug_id=8023288 and Java7 is EOL - https://www.java.com/en/download/faq/java_7.xml I think it's premature at this point to talk about timelines as we are still evaluating the work that needs to be done and any targets we gave would just be guesses.
        Hide
        alanburlison Alan Burlison added a comment -

        15778275 is the Solaris bug that has been logged to investigate adding the fts interfaces to Solaris

        Show
        alanburlison Alan Burlison added a comment - 15778275 is the Solaris bug that has been logged to investigate adding the fts interfaces to Solaris
        Hide
        alanburlison Alan Burlison added a comment -

        See also the discussion in http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201506.mbox/%3C558BCA3A.1020602%40oracle.com%3E. The use of fts(3) should be replaced by nftw(3)

        Show
        alanburlison Alan Burlison added a comment - See also the discussion in http://mail-archives.apache.org/mod_mbox/hadoop-yarn-dev/201506.mbox/%3C558BCA3A.1020602%40oracle.com%3E . The use of fts(3) should be replaced by nftw(3)
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 patch 0m 1s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions.
        0 pre-patch 5m 21s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 40s There were no new javac warning messages.
        +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 20s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        +1 yarn tests 6m 6s Tests passed in hadoop-yarn-server-nodemanager.
            21m 21s  



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12746619/YARN-3724
        Optional Tests javac unit
        git revision trunk / 06e5dd2
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8621/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8621/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8621/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8621/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 patch 0m 1s The patch file was not named according to hadoop's naming conventions. Please see https://wiki.apache.org/hadoop/HowToContribute for instructions. 0 pre-patch 5m 21s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 40s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. +1 yarn tests 6m 6s Tests passed in hadoop-yarn-server-nodemanager.     21m 21s   Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12746619/YARN-3724 Optional Tests javac unit git revision trunk / 06e5dd2 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8621/artifact/patchprocess/whitespace.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8621/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8621/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8621/console This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -



        -1 overall



        Vote Subsystem Runtime Comment
        0 pre-patch 5m 20s Pre-patch trunk compilation is healthy.
        +1 @author 0m 0s The patch does not contain any @author tags.
        -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
        +1 javac 7m 42s There were no new javac warning messages.
        +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings.
        -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix.
        +1 install 1m 20s mvn install still works.
        +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse.
        -1 yarn tests 6m 2s Tests failed in hadoop-yarn-server-nodemanager.
            21m 19s  



        Reason Tests
        Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService



        Subsystem Report/Notes
        Patch URL http://issues.apache.org/jira/secure/attachment/12746793/YARN-3724.001.patch
        Optional Tests javac unit
        git revision trunk / ee98d63
        whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8634/artifact/patchprocess/whitespace.txt
        hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8634/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8634/testReport/
        Java 1.7.0_55
        uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/8634/console

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 pre-patch 5m 20s Pre-patch trunk compilation is healthy. +1 @author 0m 0s The patch does not contain any @author tags. -1 tests included 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac 7m 42s There were no new javac warning messages. +1 release audit 0m 19s The applied patch does not increase the total number of release audit warnings. -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix. +1 install 1m 20s mvn install still works. +1 eclipse:eclipse 0m 32s The patch built with eclipse:eclipse. -1 yarn tests 6m 2s Tests failed in hadoop-yarn-server-nodemanager.     21m 19s   Reason Tests Failed unit tests hadoop.yarn.server.nodemanager.TestDeletionService Subsystem Report/Notes Patch URL http://issues.apache.org/jira/secure/attachment/12746793/YARN-3724.001.patch Optional Tests javac unit git revision trunk / ee98d63 whitespace https://builds.apache.org/job/PreCommit-YARN-Build/8634/artifact/patchprocess/whitespace.txt hadoop-yarn-server-nodemanager test log https://builds.apache.org/job/PreCommit-YARN-Build/8634/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/8634/testReport/ Java 1.7.0_55 uname Linux asf901.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Console output https://builds.apache.org/job/PreCommit-YARN-Build/8634/console This message was automatically generated.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Alan Burlison, please use the "Target Version" field to express your intention. Fix-version is exclusively used by committers when a patch gets committed.

        IAC, 2.7.1 is already done. Targeting this for 2.7.2/2.8.0.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Alan Burlison , please use the "Target Version" field to express your intention. Fix-version is exclusively used by committers when a patch gets committed. IAC, 2.7.1 is already done. Targeting this for 2.7.2/2.8.0.
        Hide
        aw Allen Wittenauer added a comment -

        +1 committed to trunk and branch-2.

        Show
        aw Allen Wittenauer added a comment - +1 committed to trunk and branch-2.
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-trunk-Commit #8704 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8704/)
        YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #8704 (See https://builds.apache.org/job/Hadoop-trunk-Commit/8704/ ) YARN-3724 . Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk #1317 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1317/)
        YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk #1317 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/1317/ ) YARN-3724 . Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #594 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/594/)
        YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #594 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/594/ ) YARN-3724 . Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #582 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/582/)
        YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)

        • hadoop-yarn-project/CHANGES.txt
        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #582 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/582/ ) YARN-3724 . Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612) hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        Hide
        hudson Hudson added a comment -

        FAILURE: Integrated in Hadoop-Hdfs-trunk #2473 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2473/)
        YARN-3724. Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612)

        • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
        • hadoop-yarn-project/CHANGES.txt
        Show
        hudson Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #2473 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/2473/ ) YARN-3724 . Use POSIX nftw(3) instead of fts(3) (Alan Burlison via aw) (aw: rev 1aa735c188a308ca608694546c595e3c51f38612) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c hadoop-yarn-project/CHANGES.txt

          People

          • Assignee:
            alanburlison Alan Burlison
            Reporter:
            malcolm kavalsky Malcolm Kavalsky
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Time Tracking

              Estimated:
              Original Estimate - 24h
              24h
              Remaining:
              Remaining Estimate - 24h
              24h
              Logged:
              Time Spent - Not Specified
              Not Specified

                Development