Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-6930

Admins should be able to explicitly enable specific LinuxContainerRuntime in the NodeManager

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-beta1, 2.8.2
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Today, in the java land, all LinuxContainerRuntimes are always enabled when using LinuxContainerExecutor and the user can simply invoke anything that he/she wants - default, docker, java-sandbox.

      We should have a way for admins to explicitly enable only specific runtimes that he/she decides for the cluster. And by default, we should have everything other than the default one disabled.

      1. YARN-6930.001.patch
        22 kB
        Shane Kumpf
      2. YARN-6930.002.patch
        24 kB
        Shane Kumpf
      3. YARN-6930.003.patch
        24 kB
        Shane Kumpf
      4. YARN-6930.004.patch
        28 kB
        Shane Kumpf
      5. YARN-6930.005.patch
        28 kB
        Shane Kumpf
      6. YARN-6930.006.patch
        28 kB
        Shane Kumpf
      7. YARN-6930.branch-2.001.patch
        24 kB
        Shane Kumpf
      8. YARN-6930.branch-2.002.patch
        23 kB
        Shane Kumpf
      9. YARN-6930.branch-2.8.001.patch
        20 kB
        Shane Kumpf
      10. YARN-6930.branch-2.8.002.patch
        13 kB
        Shane Kumpf
      11. YARN-6930.branch-2.8.2.001.patch
        13 kB
        Shane Kumpf

        Activity

        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks for the reviews and commits, Jason Lowe!

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the reviews and commits, Jason Lowe !
        Hide
        jlowe Jason Lowe added a comment -

        Thanks, Shane Kumpf! I committed this to branch-2.8 and branch-2.8.2 as well.

        Show
        jlowe Jason Lowe added a comment - Thanks, Shane Kumpf ! I committed this to branch-2.8 and branch-2.8.2 as well.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for updating the patch!

        +1 for the branch-2.8 patch. Committing this.

        Show
        jlowe Jason Lowe added a comment - Thanks for updating the patch! +1 for the branch-2.8 patch. Committing this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 16m 4s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2.8 Compile Tests
        0 mvndep 0m 20s Maven dependency ordering for branch
        +1 mvninstall 6m 36s branch-2.8 passed
        +1 compile 2m 8s branch-2.8 passed
        +1 checkstyle 0m 32s branch-2.8 passed
        +1 mvnsite 1m 20s branch-2.8 passed
        -1 findbugs 0m 47s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in branch-2.8 has 1 extant Findbugs warnings.
        +1 javadoc 1m 0s branch-2.8 passed
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 11s the patch passed
        +1 compile 2m 4s the patch passed
        +1 javac 2m 4s the patch passed
        +1 checkstyle 0m 28s the patch passed
        +1 mvnsite 1m 20s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        +1 findbugs 3m 39s the patch passed
        +1 javadoc 1m 2s the patch passed
              Other Tests
        +1 unit 0m 24s hadoop-yarn-api in the patch passed.
        +1 unit 2m 28s hadoop-yarn-common in the patch passed.
        +1 unit 10m 14s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 19s The patch does not generate ASF License warnings.
        61m 45s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c2d96dd
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12891373/YARN-6930.branch-2.8.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
        uname Linux fc55b1bfdc23 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / 108c239
        Default Java 1.7.0_151
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17850/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17850/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17850/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 16m 4s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2.8 Compile Tests 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 6m 36s branch-2.8 passed +1 compile 2m 8s branch-2.8 passed +1 checkstyle 0m 32s branch-2.8 passed +1 mvnsite 1m 20s branch-2.8 passed -1 findbugs 0m 47s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in branch-2.8 has 1 extant Findbugs warnings. +1 javadoc 1m 0s branch-2.8 passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 11s the patch passed +1 compile 2m 4s the patch passed +1 javac 2m 4s the patch passed +1 checkstyle 0m 28s the patch passed +1 mvnsite 1m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. +1 findbugs 3m 39s the patch passed +1 javadoc 1m 2s the patch passed       Other Tests +1 unit 0m 24s hadoop-yarn-api in the patch passed. +1 unit 2m 28s hadoop-yarn-common in the patch passed. +1 unit 10m 14s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 19s The patch does not generate ASF License warnings. 61m 45s Subsystem Report/Notes Docker Image:yetus/hadoop:c2d96dd JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12891373/YARN-6930.branch-2.8.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml uname Linux fc55b1bfdc23 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / 108c239 Default Java 1.7.0_151 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17850/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17850/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17850/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks for the review Jason Lowe, makes sense. Attaching a new patch with those changes.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the review Jason Lowe , makes sense. Attaching a new patch with those changes.
        Hide
        jlowe Jason Lowe added a comment - - edited

        Thanks for the 2.8 patch!

        For consistency with the trunk/branch-2 code updateEnvForWhitelistVars should not throw ContainerExecutionException. See DelegatingLinuxContainerRuntime#updateEnvForWhitelistVars in branch-2 for reference. Once that exception declaration is removed then the TestContainerLaunch changes should also drop out of the patch.

        Show
        jlowe Jason Lowe added a comment - - edited Thanks for the 2.8 patch! For consistency with the trunk/branch-2 code updateEnvForWhitelistVars should not throw ContainerExecutionException. See DelegatingLinuxContainerRuntime#updateEnvForWhitelistVars in branch-2 for reference. Once that exception declaration is removed then the TestContainerLaunch changes should also drop out of the patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 37s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
              branch-2.8 Compile Tests
        0 mvndep 0m 21s Maven dependency ordering for branch
        +1 mvninstall 9m 1s branch-2.8 passed
        +1 compile 2m 25s branch-2.8 passed
        +1 checkstyle 0m 35s branch-2.8 passed
        +1 mvnsite 1m 33s branch-2.8 passed
        -1 findbugs 0m 56s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in branch-2.8 has 1 extant Findbugs warnings.
        +1 javadoc 1m 12s branch-2.8 passed
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 15s the patch passed
        +1 compile 2m 11s the patch passed
        +1 javac 2m 11s the patch passed
        +1 checkstyle 0m 34s the patch passed
        +1 mvnsite 1m 26s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 46s the patch passed
        +1 javadoc 1m 7s the patch passed
              Other Tests
        +1 unit 0m 26s hadoop-yarn-api in the patch passed.
        +1 unit 2m 27s hadoop-yarn-common in the patch passed.
        +1 unit 9m 51s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        50m 8s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:c2d96dd
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12891283/YARN-6930.branch-2.8.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
        uname Linux 83291e37f2cc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8 / b41a702
        Default Java 1.7.0_151
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17843/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17843/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17843/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 37s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.       branch-2.8 Compile Tests 0 mvndep 0m 21s Maven dependency ordering for branch +1 mvninstall 9m 1s branch-2.8 passed +1 compile 2m 25s branch-2.8 passed +1 checkstyle 0m 35s branch-2.8 passed +1 mvnsite 1m 33s branch-2.8 passed -1 findbugs 0m 56s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in branch-2.8 has 1 extant Findbugs warnings. +1 javadoc 1m 12s branch-2.8 passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 15s the patch passed +1 compile 2m 11s the patch passed +1 javac 2m 11s the patch passed +1 checkstyle 0m 34s the patch passed +1 mvnsite 1m 26s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 46s the patch passed +1 javadoc 1m 7s the patch passed       Other Tests +1 unit 0m 26s hadoop-yarn-api in the patch passed. +1 unit 2m 27s hadoop-yarn-common in the patch passed. +1 unit 9m 51s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 50m 8s Subsystem Report/Notes Docker Image:yetus/hadoop:c2d96dd JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12891283/YARN-6930.branch-2.8.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml uname Linux 83291e37f2cc 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8 / b41a702 Default Java 1.7.0_151 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17843/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17843/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17843/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks for the review, Jason Lowe and sorry for the mistake. I'll get a patch up for branch-2.8 as well.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the review, Jason Lowe and sorry for the mistake. I'll get a patch up for branch-2.8 as well.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        The findbugs warning is unrelated to this patch. YARN-7034 addressed the warning for trunk and branch-2, but it appears this was missed in the branch-2.8 patch. I'll open a new JIRA to address this in branch-2.8 and branch-2.8.2.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - The findbugs warning is unrelated to this patch. YARN-7034 addressed the warning for trunk and branch-2, but it appears this was missed in the branch-2.8 patch. I'll open a new JIRA to address this in branch-2.8 and branch-2.8.2.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for the patch!

        In the future, please do not set the Fix Version/s field to indicate where a patch should be targeted. That field should only be updated by the committer when the changes are committed. The Target Version/s field is for specifying where changes should/need to go, and the Fix Version/s field tracks where changes have been committed. Otherwise we can't tell from the JIRA whether this has or has not been fixed in those versions.

        The 2.8.2 patch looks OK to me, but in order for this to go into branch-2.8.2 it first needs to go into branch-2.8. However the 2.8.2 patch does not apply to branch-2.8. Could you please supply a patch for branch-2.8 as well?

        Show
        jlowe Jason Lowe added a comment - Thanks for the patch! In the future, please do not set the Fix Version/s field to indicate where a patch should be targeted. That field should only be updated by the committer when the changes are committed. The Target Version/s field is for specifying where changes should/need to go, and the Fix Version/s field tracks where changes have been committed. Otherwise we can't tell from the JIRA whether this has or has not been fixed in those versions. The 2.8.2 patch looks OK to me, but in order for this to go into branch-2.8.2 it first needs to go into branch-2.8. However the 2.8.2 patch does not apply to branch-2.8. Could you please supply a patch for branch-2.8 as well?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2.8.2 Compile Tests
        0 mvndep 0m 20s Maven dependency ordering for branch
        +1 mvninstall 7m 16s branch-2.8.2 passed
        +1 compile 2m 35s branch-2.8.2 passed
        +1 checkstyle 0m 36s branch-2.8.2 passed
        +1 mvnsite 1m 34s branch-2.8.2 passed
        -1 findbugs 0m 57s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in branch-2.8.2 has 1 extant Findbugs warnings.
        +1 javadoc 1m 16s branch-2.8.2 passed
              Patch Compile Tests
        0 mvndep 0m 12s Maven dependency ordering for patch
        +1 mvninstall 1m 29s the patch passed
        +1 compile 2m 35s the patch passed
        +1 javac 2m 35s the patch passed
        +1 checkstyle 0m 36s the patch passed
        +1 mvnsite 1m 36s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 4m 26s the patch passed
        +1 javadoc 1m 16s the patch passed
              Other Tests
        +1 unit 0m 30s hadoop-yarn-api in the patch passed.
        +1 unit 2m 40s hadoop-yarn-common in the patch passed.
        +1 unit 9m 45s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 20s The patch does not generate ASF License warnings.
        50m 31s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:d0a0f24
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12891051/YARN-6930.branch-2.8.2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml
        uname Linux b2cd08fe04dd 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2.8.2 / 76e053a
        Default Java 1.7.0_151
        findbugs v3.0.0
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17824/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17824/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17824/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2.8.2 Compile Tests 0 mvndep 0m 20s Maven dependency ordering for branch +1 mvninstall 7m 16s branch-2.8.2 passed +1 compile 2m 35s branch-2.8.2 passed +1 checkstyle 0m 36s branch-2.8.2 passed +1 mvnsite 1m 34s branch-2.8.2 passed -1 findbugs 0m 57s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in branch-2.8.2 has 1 extant Findbugs warnings. +1 javadoc 1m 16s branch-2.8.2 passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 2m 35s the patch passed +1 javac 2m 35s the patch passed +1 checkstyle 0m 36s the patch passed +1 mvnsite 1m 36s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 4m 26s the patch passed +1 javadoc 1m 16s the patch passed       Other Tests +1 unit 0m 30s hadoop-yarn-api in the patch passed. +1 unit 2m 40s hadoop-yarn-common in the patch passed. +1 unit 9m 45s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 20s The patch does not generate ASF License warnings. 50m 31s Subsystem Report/Notes Docker Image:yetus/hadoop:d0a0f24 JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12891051/YARN-6930.branch-2.8.2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle xml uname Linux b2cd08fe04dd 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2.8.2 / 76e053a Default Java 1.7.0_151 findbugs v3.0.0 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17824/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17824/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17824/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        This is needed in 2.8.2 as well. Reopened and attached a patch for 2.8.2.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - This is needed in 2.8.2 as well. Reopened and attached a patch for 2.8.2.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks Jason Lowe! Also, thanks to Eric Badger and Miklos Szegedi for the reviews!

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks Jason Lowe ! Also, thanks to Eric Badger and Miklos Szegedi for the reviews!
        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12812 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12812/)
        YARN-6930. Admins should be able to explicitly enable specific (jlowe: rev b0b535d9d5727cd84fd6368c6d1b38363616504e)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DelegatingLinuxContainerRuntime.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/LinuxContainerRuntimeConstants.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/runtime/ContainerRuntime.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDelegatingLinuxContainerRuntime.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12812 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12812/ ) YARN-6930 . Admins should be able to explicitly enable specific (jlowe: rev b0b535d9d5727cd84fd6368c6d1b38363616504e) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/monitor/ContainersMonitorImpl.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DelegatingLinuxContainerRuntime.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/LinuxContainerRuntimeConstants.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/runtime/ContainerRuntime.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDelegatingLinuxContainerRuntime.java
        Hide
        jlowe Jason Lowe added a comment -

        Thanks to Shane Kumpf for the contribution and to Miklos Szegedi and Eric Badger for additional review! I committed this to trunk, branch-3.0, and branch-2.

        Show
        jlowe Jason Lowe added a comment - Thanks to Shane Kumpf for the contribution and to Miklos Szegedi and Eric Badger for additional review! I committed this to trunk, branch-3.0, and branch-2.
        Hide
        jlowe Jason Lowe added a comment -

        +1 for the branch-2 patch as well. The TestNMWebServer failure is unrelated as the same failure occurs without the patch applied. I'll file a followup JIRA to address that test failure in branch-2 if one doesn't already exist.

        Committing this.

        Show
        jlowe Jason Lowe added a comment - +1 for the branch-2 patch as well. The TestNMWebServer failure is unrelated as the same failure occurs without the patch applied. I'll file a followup JIRA to address that test failure in branch-2 if one doesn't already exist. Committing this.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 31s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        0 mvndep 0m 12s Maven dependency ordering for branch
        +1 mvninstall 8m 0s branch-2 passed
        +1 compile 2m 3s branch-2 passed with JDK v1.8.0_144
        +1 compile 2m 17s branch-2 passed with JDK v1.7.0_151
        +1 checkstyle 0m 42s branch-2 passed
        +1 mvnsite 1m 49s branch-2 passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 3m 21s branch-2 passed
        +1 javadoc 1m 15s branch-2 passed with JDK v1.8.0_144
        +1 javadoc 1m 26s branch-2 passed with JDK v1.7.0_151
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 28s the patch passed
        +1 compile 1m 53s the patch passed with JDK v1.8.0_144
        +1 javac 1m 53s the patch passed
        +1 compile 2m 17s the patch passed with JDK v1.7.0_151
        +1 javac 2m 17s the patch passed
        +1 checkstyle 0m 41s the patch passed
        +1 mvnsite 1m 44s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 3m 53s the patch passed
        +1 javadoc 0m 16s hadoop-yarn-api in the patch passed with JDK v1.8.0_144.
        +1 javadoc 0m 26s hadoop-yarn-common in the patch passed with JDK v1.8.0_144.
        +1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_144 with JDK v1.8.0_144 generated 0 new + 201 unchanged - 1 fixed = 201 total (was 202)
        +1 javadoc 0m 9s hadoop-yarn-site in the patch passed with JDK v1.8.0_144.
        +1 javadoc 1m 19s the patch passed with JDK v1.7.0_151
              Other Tests
        +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_151.
        +1 unit 2m 26s hadoop-yarn-common in the patch passed with JDK v1.7.0_151.
        -1 unit 13m 49s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_151.
        +1 unit 0m 9s hadoop-yarn-site in the patch passed with JDK v1.7.0_151.
        +1 asflicense 0m 21s The patch does not generate ASF License warnings.
        77m 21s



        Reason Tests
        JDK v1.8.0_144 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
        JDK v1.7.0_151 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885825/YARN-6930.branch-2.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 4e2d746e28bd 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 8cce192
        Default Java 1.7.0_151
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151
        findbugs v3.0.0
        unit https://builds.apache.org/job/PreCommit-YARN-Build/17331/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_151.txt
        JDK v1.7.0_151 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17331/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17331/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 31s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 12s Maven dependency ordering for branch +1 mvninstall 8m 0s branch-2 passed +1 compile 2m 3s branch-2 passed with JDK v1.8.0_144 +1 compile 2m 17s branch-2 passed with JDK v1.7.0_151 +1 checkstyle 0m 42s branch-2 passed +1 mvnsite 1m 49s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 21s branch-2 passed +1 javadoc 1m 15s branch-2 passed with JDK v1.8.0_144 +1 javadoc 1m 26s branch-2 passed with JDK v1.7.0_151       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 28s the patch passed +1 compile 1m 53s the patch passed with JDK v1.8.0_144 +1 javac 1m 53s the patch passed +1 compile 2m 17s the patch passed with JDK v1.7.0_151 +1 javac 2m 17s the patch passed +1 checkstyle 0m 41s the patch passed +1 mvnsite 1m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 53s the patch passed +1 javadoc 0m 16s hadoop-yarn-api in the patch passed with JDK v1.8.0_144. +1 javadoc 0m 26s hadoop-yarn-common in the patch passed with JDK v1.8.0_144. +1 javadoc 0m 16s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_144 with JDK v1.8.0_144 generated 0 new + 201 unchanged - 1 fixed = 201 total (was 202) +1 javadoc 0m 9s hadoop-yarn-site in the patch passed with JDK v1.8.0_144. +1 javadoc 1m 19s the patch passed with JDK v1.7.0_151       Other Tests +1 unit 0m 27s hadoop-yarn-api in the patch passed with JDK v1.7.0_151. +1 unit 2m 26s hadoop-yarn-common in the patch passed with JDK v1.7.0_151. -1 unit 13m 49s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_151. +1 unit 0m 9s hadoop-yarn-site in the patch passed with JDK v1.7.0_151. +1 asflicense 0m 21s The patch does not generate ASF License warnings. 77m 21s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer JDK v1.7.0_151 Failed junit tests hadoop.yarn.server.nodemanager.webapp.TestNMWebServer Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885825/YARN-6930.branch-2.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 4e2d746e28bd 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 8cce192 Default Java 1.7.0_151 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/17331/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_151.txt JDK v1.7.0_151 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17331/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17331/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Attached a patch to address the checkstyle issue. The test failures appear unrelated.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Attached a patch to address the checkstyle issue. The test failures appear unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 39s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              branch-2 Compile Tests
        0 mvndep 0m 30s Maven dependency ordering for branch
        +1 mvninstall 7m 9s branch-2 passed
        +1 compile 2m 0s branch-2 passed with JDK v1.8.0_144
        +1 compile 2m 21s branch-2 passed with JDK v1.7.0_151
        +1 checkstyle 0m 43s branch-2 passed
        +1 mvnsite 1m 49s branch-2 passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 3m 27s branch-2 passed
        +1 javadoc 1m 14s branch-2 passed with JDK v1.8.0_144
        +1 javadoc 1m 25s branch-2 passed with JDK v1.7.0_151
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 32s the patch passed
        +1 compile 1m 59s the patch passed with JDK v1.8.0_144
        +1 javac 1m 59s the patch passed
        +1 compile 2m 20s the patch passed with JDK v1.7.0_151
        +1 javac 2m 20s the patch passed
        -0 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 239 unchanged - 0 fixed = 240 total (was 239)
        +1 mvnsite 1m 47s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 0s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 4m 57s the patch passed
        +1 javadoc 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_144.
        +1 javadoc 0m 38s hadoop-yarn-common in the patch passed with JDK v1.8.0_144.
        +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_144 with JDK v1.8.0_144 generated 0 new + 201 unchanged - 1 fixed = 201 total (was 202)
        +1 javadoc 0m 10s hadoop-yarn-site in the patch passed with JDK v1.8.0_144.
        +1 javadoc 1m 42s the patch passed with JDK v1.7.0_151
              Other Tests
        +1 unit 0m 33s hadoop-yarn-api in the patch passed with JDK v1.7.0_151.
        +1 unit 2m 53s hadoop-yarn-common in the patch passed with JDK v1.7.0_151.
        -1 unit 28m 40s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_151.
        +1 unit 0m 14s hadoop-yarn-site in the patch passed with JDK v1.7.0_151.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        97m 12s



        Reason Tests
        JDK v1.8.0_144 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing
          hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
        JDK v1.7.0_151 Failed junit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels
          hadoop.yarn.server.nodemanager.webapp.TestNMWebServer
        JDK v1.7.0_151 Timed out junit tests org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:5e40efe
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885811/YARN-6930.branch-2.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 0bc436afe1a1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision branch-2 / 8cce192
        Default Java 1.7.0_151
        Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17327/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/17327/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_151.txt
        JDK v1.7.0_151 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17327/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17327/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 39s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 30s Maven dependency ordering for branch +1 mvninstall 7m 9s branch-2 passed +1 compile 2m 0s branch-2 passed with JDK v1.8.0_144 +1 compile 2m 21s branch-2 passed with JDK v1.7.0_151 +1 checkstyle 0m 43s branch-2 passed +1 mvnsite 1m 49s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 27s branch-2 passed +1 javadoc 1m 14s branch-2 passed with JDK v1.8.0_144 +1 javadoc 1m 25s branch-2 passed with JDK v1.7.0_151       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 32s the patch passed +1 compile 1m 59s the patch passed with JDK v1.8.0_144 +1 javac 1m 59s the patch passed +1 compile 2m 20s the patch passed with JDK v1.7.0_151 +1 javac 2m 20s the patch passed -0 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 239 unchanged - 0 fixed = 240 total (was 239) +1 mvnsite 1m 47s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 0s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 4m 57s the patch passed +1 javadoc 0m 22s hadoop-yarn-api in the patch passed with JDK v1.8.0_144. +1 javadoc 0m 38s hadoop-yarn-common in the patch passed with JDK v1.8.0_144. +1 javadoc 0m 21s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_144 with JDK v1.8.0_144 generated 0 new + 201 unchanged - 1 fixed = 201 total (was 202) +1 javadoc 0m 10s hadoop-yarn-site in the patch passed with JDK v1.8.0_144. +1 javadoc 1m 42s the patch passed with JDK v1.7.0_151       Other Tests +1 unit 0m 33s hadoop-yarn-api in the patch passed with JDK v1.7.0_151. +1 unit 2m 53s hadoop-yarn-common in the patch passed with JDK v1.7.0_151. -1 unit 28m 40s hadoop-yarn-server-nodemanager in the patch failed with JDK v1.7.0_151. +1 unit 0m 14s hadoop-yarn-site in the patch passed with JDK v1.7.0_151. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 97m 12s Reason Tests JDK v1.8.0_144 Failed junit tests hadoop.yarn.server.nodemanager.containermanager.scheduler.TestContainerSchedulerQueuing   hadoop.yarn.server.nodemanager.webapp.TestNMWebServer JDK v1.7.0_151 Failed junit tests hadoop.yarn.server.nodemanager.TestNodeStatusUpdaterForLabels   hadoop.yarn.server.nodemanager.webapp.TestNMWebServer JDK v1.7.0_151 Timed out junit tests org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater Subsystem Report/Notes Docker Image:yetus/hadoop:5e40efe JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885811/YARN-6930.branch-2.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 0bc436afe1a1 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 8cce192 Default Java 1.7.0_151 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_144 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_151 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17327/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17327/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_151.txt JDK v1.7.0_151 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17327/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17327/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Sorry for the delay. I'm attaching a patch for branch-2. Note that the java sandbox runtime (YARN-5280) does not exist in branch-2. Without YARN-5280, there is no call to prepareContainer in ContainerLaunch, so how the application fails differs. However, the application is properly failed when the runtime is not allowed, which is the goal here.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Sorry for the delay. I'm attaching a patch for branch-2. Note that the java sandbox runtime ( YARN-5280 ) does not exist in branch-2. Without YARN-5280 , there is no call to prepareContainer in ContainerLaunch, so how the application fails differs. However, the application is properly failed when the runtime is not allowed, which is the goal here.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for updating the patch! +1 lgtm as well. Shane Kumpf would you mind providing a patch for branch-2 as well?

        Show
        jlowe Jason Lowe added a comment - Thanks for updating the patch! +1 lgtm as well. Shane Kumpf would you mind providing a patch for branch-2 as well?
        Hide
        ebadger Eric Badger added a comment -

        +1 (non-binding) from me on the latest patch

        Show
        ebadger Eric Badger added a comment - +1 (non-binding) from me on the latest patch
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 19s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 42s Maven dependency ordering for branch
        +1 mvninstall 14m 0s trunk passed
        +1 compile 9m 26s trunk passed
        +1 checkstyle 1m 1s trunk passed
        +1 mvnsite 2m 18s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 48s trunk passed
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 30s the patch passed
        +1 compile 5m 53s the patch passed
        +1 javac 5m 53s the patch passed
        +1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 215 unchanged - 1 fixed = 215 total (was 216)
        +1 mvnsite 2m 9s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 3m 46s the patch passed
        +1 javadoc 0m 25s hadoop-yarn-api in the patch passed.
        +1 javadoc 0m 44s hadoop-yarn-common in the patch passed.
        +1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105)
        +1 javadoc 0m 15s hadoop-yarn-site in the patch passed.
              Other Tests
        +1 unit 0m 35s hadoop-yarn-api in the patch passed.
        +1 unit 2m 33s hadoop-yarn-common in the patch passed.
        +1 unit 13m 36s hadoop-yarn-server-nodemanager in the patch passed.
        +1 unit 0m 15s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        75m 3s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884476/YARN-6930.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux cfe947ce94f8 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 200b113
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17206/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17206/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17206/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 42s Maven dependency ordering for branch +1 mvninstall 14m 0s trunk passed +1 compile 9m 26s trunk passed +1 checkstyle 1m 1s trunk passed +1 mvnsite 2m 18s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 48s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 30s the patch passed +1 compile 5m 53s the patch passed +1 javac 5m 53s the patch passed +1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 215 unchanged - 1 fixed = 215 total (was 216) +1 mvnsite 2m 9s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 46s the patch passed +1 javadoc 0m 25s hadoop-yarn-api in the patch passed. +1 javadoc 0m 44s hadoop-yarn-common in the patch passed. +1 javadoc 0m 28s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105) +1 javadoc 0m 15s hadoop-yarn-site in the patch passed.       Other Tests +1 unit 0m 35s hadoop-yarn-api in the patch passed. +1 unit 2m 33s hadoop-yarn-common in the patch passed. +1 unit 13m 36s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 15s hadoop-yarn-site in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 75m 3s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884476/YARN-6930.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux cfe947ce94f8 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 200b113 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17206/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17206/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17206/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks for the additional reviews. I've attached a new patch to address the comments.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the additional reviews. I've attached a new patch to address the comments.
        Hide
        jlowe Jason Lowe added a comment -

        Thanks for updating the patch!

        In addition to Eric's comment, it would be nice if we identified which word in the config value wasn't recognized in this error handling. The code should be catching inside the for loop so it knows which value was bad and can report it.

            try {
              for (String configuredRuntime : configuredRuntimes) {
                allowedRuntimes.add(
                    LinuxContainerRuntimeType.valueOf(configuredRuntime.toUpperCase()));
              }
            } catch (IllegalArgumentException e) {
              throw new ContainerExecutionException("Invalid runtime configuration for "
                  + LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES);
            }
        

        Otherwise patch looks good overall.

        Show
        jlowe Jason Lowe added a comment - Thanks for updating the patch! In addition to Eric's comment, it would be nice if we identified which word in the config value wasn't recognized in this error handling. The code should be catching inside the for loop so it knows which value was bad and can report it. try { for ( String configuredRuntime : configuredRuntimes) { allowedRuntimes.add( LinuxContainerRuntimeType.valueOf(configuredRuntime.toUpperCase())); } } catch (IllegalArgumentException e) { throw new ContainerExecutionException( "Invalid runtime configuration for " + LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES); } Otherwise patch looks good overall.
        Hide
        ebadger Eric Badger added a comment - - edited
        +  <property>
        +    <name>yarn.nodemanager.runtime.linux.allowed-runtimes</name>
        +    <value>DefaultLinuxContainerRuntime,DockerLinuxContainerRuntime</value>
        +    <description>
        +      Comma separated list of runtimes that are allowed when using
        +      LinuxContainerExecutor.
        +    </description>
        +  </property>
        

        We need to update this with the new enums.

        Also, do we need a new file for the enum? Can we use LinuxContainerRuntimeConstants.java instead?

        Show
        ebadger Eric Badger added a comment - - edited + <property> + <name>yarn.nodemanager.runtime.linux.allowed-runtimes</name> + <value>DefaultLinuxContainerRuntime,DockerLinuxContainerRuntime</value> + <description> + Comma separated list of runtimes that are allowed when using + LinuxContainerExecutor. + </description> + </property> We need to update this with the new enums. Also, do we need a new file for the enum? Can we use LinuxContainerRuntimeConstants.java instead?
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 46s Maven dependency ordering for branch
        +1 mvninstall 16m 42s trunk passed
        +1 compile 9m 31s trunk passed
        +1 checkstyle 0m 58s trunk passed
        +1 mvnsite 2m 7s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 47s trunk passed
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 29s the patch passed
        +1 compile 5m 38s the patch passed
        +1 javac 5m 38s the patch passed
        +1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 1 fixed = 214 total (was 215)
        +1 mvnsite 2m 4s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 3m 29s the patch passed
        +1 javadoc 0m 23s hadoop-yarn-api in the patch passed.
        +1 javadoc 0m 42s hadoop-yarn-common in the patch passed.
        +1 javadoc 0m 23s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105)
        +1 javadoc 0m 16s hadoop-yarn-site in the patch passed.
              Other Tests
        +1 unit 0m 32s hadoop-yarn-api in the patch passed.
        +1 unit 2m 28s hadoop-yarn-common in the patch passed.
        +1 unit 13m 33s hadoop-yarn-server-nodemanager in the patch passed.
        +1 unit 0m 16s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 27s The patch does not generate ASF License warnings.
        76m 27s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884245/YARN-6930.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 67e2572c4009 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 33afa94
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17179/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17179/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17179/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 46s Maven dependency ordering for branch +1 mvninstall 16m 42s trunk passed +1 compile 9m 31s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 2m 7s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 47s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 29s the patch passed +1 compile 5m 38s the patch passed +1 javac 5m 38s the patch passed +1 checkstyle 0m 58s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 214 unchanged - 1 fixed = 214 total (was 215) +1 mvnsite 2m 4s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 29s the patch passed +1 javadoc 0m 23s hadoop-yarn-api in the patch passed. +1 javadoc 0m 42s hadoop-yarn-common in the patch passed. +1 javadoc 0m 23s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105) +1 javadoc 0m 16s hadoop-yarn-site in the patch passed.       Other Tests +1 unit 0m 32s hadoop-yarn-api in the patch passed. +1 unit 2m 28s hadoop-yarn-common in the patch passed. +1 unit 13m 33s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 16s hadoop-yarn-site in the patch passed. +1 asflicense 0m 27s The patch does not generate ASF License warnings. 76m 27s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884245/YARN-6930.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 67e2572c4009 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 33afa94 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17179/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17179/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17179/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        I'll upload a new patch to address the checkstyle issue. The test that failed passes locally and doesn't appear to be related to this patch.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - I'll upload a new patch to address the checkstyle issue. The test that failed passes locally and doesn't appear to be related to this patch.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 20s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 15m 44s trunk passed
        +1 compile 10m 11s trunk passed
        +1 checkstyle 1m 3s trunk passed
        +1 mvnsite 2m 16s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 55s trunk passed
              Patch Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 36s the patch passed
        +1 compile 6m 14s the patch passed
        +1 javac 6m 14s the patch passed
        -0 checkstyle 1m 2s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 214 unchanged - 1 fixed = 215 total (was 215)
        +1 mvnsite 2m 13s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 3m 57s the patch passed
        +1 javadoc 0m 28s hadoop-yarn-api in the patch passed.
        +1 javadoc 0m 44s hadoop-yarn-common in the patch passed.
        +1 javadoc 0m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105)
        +1 javadoc 0m 15s hadoop-yarn-site in the patch passed.
              Other Tests
        +1 unit 0m 36s hadoop-yarn-api in the patch passed.
        +1 unit 2m 42s hadoop-yarn-common in the patch passed.
        -1 unit 13m 53s hadoop-yarn-server-nodemanager in the patch failed.
        +1 unit 0m 14s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        78m 24s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884091/YARN-6930.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 113027a99b13 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / d5bc958
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17154/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17154/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/17154/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17154/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/17154/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 15m 44s trunk passed +1 compile 10m 11s trunk passed +1 checkstyle 1m 3s trunk passed +1 mvnsite 2m 16s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 55s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 36s the patch passed +1 compile 6m 14s the patch passed +1 javac 6m 14s the patch passed -0 checkstyle 1m 2s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 214 unchanged - 1 fixed = 215 total (was 215) +1 mvnsite 2m 13s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 57s the patch passed +1 javadoc 0m 28s hadoop-yarn-api in the patch passed. +1 javadoc 0m 44s hadoop-yarn-common in the patch passed. +1 javadoc 0m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105) +1 javadoc 0m 15s hadoop-yarn-site in the patch passed.       Other Tests +1 unit 0m 36s hadoop-yarn-api in the patch passed. +1 unit 2m 42s hadoop-yarn-common in the patch passed. -1 unit 13m 53s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 0m 14s hadoop-yarn-site in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 78m 24s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.TestContainerManager Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884091/YARN-6930.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 113027a99b13 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d5bc958 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17154/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17154/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17154/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17154/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17154/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks for the feedback Jason Lowe and Eric Badger. I have uploaded a new patch to address the comments. Runtime types are now represented as an enum, docs have been updated with allowed values, and the runtimes will no longer be instantiated or initialized if the runtime is not allowed.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the feedback Jason Lowe and Eric Badger . I have uploaded a new patch to address the comments. Runtime types are now represented as an enum, docs have been updated with allowed values, and the runtimes will no longer be instantiated or initialized if the runtime is not allowed.
        Hide
        jlowe Jason Lowe added a comment -

        I'm not hung up on whether we go full-blown enums, but I do think we should not do "semi-reflection" with the base class names. IMHO we should either do a full plugin interface with reflection or not use reflection at all. Maybe others feel differently.

        When you say "list the ones we know about here", can you elaborate on what you mean?

        Sorry, I meant in the property description for yarn.nodemanager.runtime.linux.allowed-runtimes in yarn-default, which last I checked was used to automatically generate the property documentation in the Hadoop docs. If a user stumbles across this property, the description is not going to be sufficient to know what valid values are allowed.

        Show
        jlowe Jason Lowe added a comment - I'm not hung up on whether we go full-blown enums, but I do think we should not do "semi-reflection" with the base class names. IMHO we should either do a full plugin interface with reflection or not use reflection at all. Maybe others feel differently. When you say "list the ones we know about here", can you elaborate on what you mean? Sorry, I meant in the property description for yarn.nodemanager.runtime.linux.allowed-runtimes in yarn-default, which last I checked was used to automatically generate the property documentation in the Hadoop docs. If a user stumbles across this property, the description is not going to be sufficient to know what valid values are allowed.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks for the feedback, Jason Lowe! I'm not opposed to moving this to enums and simpler names if that is the preferred direction.

        If we are going the hardcoded route then arguably we should list the ones we know about here so users know what they can do.

        When you say "list the ones we know about here", can you elaborate on what you mean? List them in the documentation?

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the feedback, Jason Lowe ! I'm not opposed to moving this to enums and simpler names if that is the preferred direction. If we are going the hardcoded route then arguably we should list the ones we know about here so users know what they can do. When you say "list the ones we know about here", can you elaborate on what you mean? List them in the documentation?
        Hide
        jlowe Jason Lowe added a comment -

        Sorry to jump in a bit late here. I agree that going the reflection route is a bit dubious at this point for a couple of reasons. This isn't a supported plugin interface yet, meaning we don't expect users to provide their own runtime classes which would require reflection to load. Also the runtime interface would need at least a method to determine whether that runtime wants to support a particular container. Currently DelegatingLinuxContainerRuntime#pickContainerRuntime has very specific knowledge of all the various runtimes and is doing that logic. It would need to be changed to an ordered list where the first runtime to say it wants to support a container gets it. So I think we're OK keeping it hardcoded for now given it's not really a general interface yet. Even if we want to migrate to a full plugin interface later that requires fully-qualified class names in the property list, we can always continue to recognize the legacy values for backwards compatibility.

        On a related note, how does a user know what values are valid for yarn.nodemanager.runtime.linux.allowed-runtimes? If we are going the hardcoded route then arguably we should list the ones we know about here so users know what they can do. Also from a performance standpoint, if we're going hardcoded then it would be more efficient to use enums for the various supported runtimes and use an EnumSet to track allowed runtimes. Then we aren't doing a linear search with string compares for each runtime on each container operation but rather a simple bitmask check. Not a must-fix thing, just pointing out the possibility.

        We could also explicitly decouple the config values from the source, using simpler values like "default,docker,javasandbox" rather than the class name. The latter has the drawback that if anyone decides to refactor the class name they just inadvertently broke all existing user configs that referred to it. If we keep an explicit mapping of values to classnames (like the enum) then this can't happen.

        Show
        jlowe Jason Lowe added a comment - Sorry to jump in a bit late here. I agree that going the reflection route is a bit dubious at this point for a couple of reasons. This isn't a supported plugin interface yet, meaning we don't expect users to provide their own runtime classes which would require reflection to load. Also the runtime interface would need at least a method to determine whether that runtime wants to support a particular container. Currently DelegatingLinuxContainerRuntime#pickContainerRuntime has very specific knowledge of all the various runtimes and is doing that logic. It would need to be changed to an ordered list where the first runtime to say it wants to support a container gets it. So I think we're OK keeping it hardcoded for now given it's not really a general interface yet. Even if we want to migrate to a full plugin interface later that requires fully-qualified class names in the property list, we can always continue to recognize the legacy values for backwards compatibility. On a related note, how does a user know what values are valid for yarn.nodemanager.runtime.linux.allowed-runtimes? If we are going the hardcoded route then arguably we should list the ones we know about here so users know what they can do. Also from a performance standpoint, if we're going hardcoded then it would be more efficient to use enums for the various supported runtimes and use an EnumSet to track allowed runtimes. Then we aren't doing a linear search with string compares for each runtime on each container operation but rather a simple bitmask check. Not a must-fix thing, just pointing out the possibility. We could also explicitly decouple the config values from the source, using simpler values like "default,docker,javasandbox" rather than the class name. The latter has the drawback that if anyone decides to refactor the class name they just inadvertently broke all existing user configs that referred to it. If we keep an explicit mapping of values to classnames (like the enum) then this can't happen.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks for the review, Eric Badger.

        The findbugs isn't related to this JIRA, butI think it's something that we should fix.

        See YARN-6515. YARN-6968 has been created to track fixing the issue, but I expect we'll remove the reference all together once whitelisted mounts are available.

        Is it ok to import static variables one by one or should we just import YarnConfiguration and reference the variables specifically?

        Given line length limits and how long some of the config options have become, IMO, it improves readability, but I'll let others comment if there is a strong opinion.

        I also wonder whether it would be better/cleaner to iterate through allowedRuntimes and allocate the new objects by class name via reflection instead of hard-coding all of them

        I've always tried to reserve reflection for cases where I don't know a class and its functionality at compile time. In this case we do. Again, if there is a strong opinion that the trade off is worth it, I can pursue a patch that leverages reflection.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the review, Eric Badger . The findbugs isn't related to this JIRA, butI think it's something that we should fix. See YARN-6515 . YARN-6968 has been created to track fixing the issue, but I expect we'll remove the reference all together once whitelisted mounts are available. Is it ok to import static variables one by one or should we just import YarnConfiguration and reference the variables specifically? Given line length limits and how long some of the config options have become, IMO, it improves readability, but I'll let others comment if there is a strong opinion. I also wonder whether it would be better/cleaner to iterate through allowedRuntimes and allocate the new objects by class name via reflection instead of hard-coding all of them I've always tried to reserve reflection for cases where I don't know a class and its functionality at compile time. In this case we do. Again, if there is a strong opinion that the trade off is worth it, I can pursue a patch that leverages reflection.
        Hide
        ebadger Eric Badger added a comment -

        File YARN-7025 for the findbugs issue

        Show
        ebadger Eric Badger added a comment - File YARN-7025 for the findbugs issue
        Hide
        ebadger Eric Badger added a comment -

        The findbugs isn't related to this JIRA, butI think it's something that we should fix. We can just make CGROUPS_ROOT_DIRECTORY another NM config.

        DelegatingLinuxContainerRuntime.java

        @@ -31,8 +32,13 @@
         import org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerRuntime;
         import org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerRuntimeContext;
         
        +import java.util.Arrays;
        +import java.util.List;
         import java.util.Map;
         
        +import static org.apache.hadoop.yarn.conf.YarnConfiguration.DEFAULT_LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES;
        +import static org.apache.hadoop.yarn.conf.YarnConfiguration.LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES;
        +
        

        Not sure of the convention here. Is it ok to import static variables one by one or should we just import YarnConfiguration and reference the variables specifically?

        @@ -50,25 +56,35 @@
           private DefaultLinuxContainerRuntime defaultLinuxContainerRuntime;
           private DockerLinuxContainerRuntime dockerLinuxContainerRuntime;
           private JavaSandboxLinuxContainerRuntime javaSandboxLinuxContainerRuntime;
        +  private List<String> allowedRuntimes;
         
           @Override
           public void initialize(Configuration conf)
               throws ContainerExecutionException {
        +    allowedRuntimes = Arrays.asList(
        +        conf.getTrimmedStrings(LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES,
        +        DEFAULT_LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES));
             PrivilegedOperationExecutor privilegedOperationExecutor =
                 PrivilegedOperationExecutor.getInstance(conf);
             defaultLinuxContainerRuntime = new DefaultLinuxContainerRuntime(
                 privilegedOperationExecutor);
        -    defaultLinuxContainerRuntime.initialize(conf);
        +    if (isRuntimeAllowed(defaultLinuxContainerRuntime)) {
        +      defaultLinuxContainerRuntime.initialize(conf);
        +    }
             dockerLinuxContainerRuntime = new DockerLinuxContainerRuntime(
                 privilegedOperationExecutor);
        -    dockerLinuxContainerRuntime.initialize(conf);
        +    if (isRuntimeAllowed(dockerLinuxContainerRuntime)) {
        +      dockerLinuxContainerRuntime.initialize(conf);
        +    }
             javaSandboxLinuxContainerRuntime = new JavaSandboxLinuxContainerRuntime(
                 privilegedOperationExecutor);
        -    javaSandboxLinuxContainerRuntime.initialize(conf);
        +    if (isRuntimeAllowed(javaSandboxLinuxContainerRuntime)) {
        +      javaSandboxLinuxContainerRuntime.initialize(conf);
        +    }
        

        It would save us a little bit of time and memory isRuntimeAllowed took the class name directly. That way we could check to see if the runtime is allowed before we allocate it. I also wonder whether it would be better/cleaner to iterate through allowedRuntimes and allocate the new objects by class name via reflection instead of hard-coding all of them. There's probably a reason why this is a terrible idea, but it's a thought.

        Show
        ebadger Eric Badger added a comment - The findbugs isn't related to this JIRA, butI think it's something that we should fix. We can just make CGROUPS_ROOT_DIRECTORY another NM config. DelegatingLinuxContainerRuntime.java @@ -31,8 +32,13 @@ import org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerRuntime; import org.apache.hadoop.yarn.server.nodemanager.containermanager.runtime.ContainerRuntimeContext; +import java.util.Arrays; +import java.util.List; import java.util.Map; +import static org.apache.hadoop.yarn.conf.YarnConfiguration.DEFAULT_LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES; +import static org.apache.hadoop.yarn.conf.YarnConfiguration.LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES; + Not sure of the convention here. Is it ok to import static variables one by one or should we just import YarnConfiguration and reference the variables specifically? @@ -50,25 +56,35 @@ private DefaultLinuxContainerRuntime defaultLinuxContainerRuntime; private DockerLinuxContainerRuntime dockerLinuxContainerRuntime; private JavaSandboxLinuxContainerRuntime javaSandboxLinuxContainerRuntime; + private List<String> allowedRuntimes; @Override public void initialize(Configuration conf) throws ContainerExecutionException { + allowedRuntimes = Arrays.asList( + conf.getTrimmedStrings(LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES, + DEFAULT_LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES)); PrivilegedOperationExecutor privilegedOperationExecutor = PrivilegedOperationExecutor.getInstance(conf); defaultLinuxContainerRuntime = new DefaultLinuxContainerRuntime( privilegedOperationExecutor); - defaultLinuxContainerRuntime.initialize(conf); + if (isRuntimeAllowed(defaultLinuxContainerRuntime)) { + defaultLinuxContainerRuntime.initialize(conf); + } dockerLinuxContainerRuntime = new DockerLinuxContainerRuntime( privilegedOperationExecutor); - dockerLinuxContainerRuntime.initialize(conf); + if (isRuntimeAllowed(dockerLinuxContainerRuntime)) { + dockerLinuxContainerRuntime.initialize(conf); + } javaSandboxLinuxContainerRuntime = new JavaSandboxLinuxContainerRuntime( privilegedOperationExecutor); - javaSandboxLinuxContainerRuntime.initialize(conf); + if (isRuntimeAllowed(javaSandboxLinuxContainerRuntime)) { + javaSandboxLinuxContainerRuntime.initialize(conf); + } It would save us a little bit of time and memory isRuntimeAllowed took the class name directly. That way we could check to see if the runtime is allowed before we allocate it. I also wonder whether it would be better/cleaner to iterate through allowedRuntimes and allocate the new objects by class name via reflection instead of hard-coding all of them. There's probably a reason why this is a terrible idea, but it's a thought.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        The unit test failure does not appear to be related to this patch. It is failing on trunk today. I've opened YARN-7014 to investigate. I believe the latest patch has addressed the comments. /cc Jian He

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - The unit test failure does not appear to be related to this patch. It is failing on trunk today. I've opened YARN-7014 to investigate. I believe the latest patch has addressed the comments. /cc Jian He
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks Miklos Szegedi and Eric Badger for the feedback. I have attached a new patch that addresses your comments and the issues reported by the precommit job.

        Should we set runtime = null if the runtime isn't allowed, just in case someone catches the ContainerExecutionException somewhere up the line?

        I don't believe this is necessary given the scope of runtime, so I've left this change out. Let me know if I'm missing something here.

        I'm still looking into the c-e test failure, but it may be unrelated.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks Miklos Szegedi and Eric Badger for the feedback. I have attached a new patch that addresses your comments and the issues reported by the precommit job. Should we set runtime = null if the runtime isn't allowed, just in case someone catches the ContainerExecutionException somewhere up the line? I don't believe this is necessary given the scope of runtime, so I've left this change out. Let me know if I'm missing something here. I'm still looking into the c-e test failure, but it may be unrelated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 24s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 16m 5s trunk passed
        +1 compile 9m 20s trunk passed
        +1 checkstyle 1m 0s trunk passed
        +1 mvnsite 2m 12s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 47s trunk passed
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 35s the patch passed
        +1 compile 6m 8s the patch passed
        +1 javac 6m 8s the patch passed
        +1 checkstyle 1m 4s the patch passed
        +1 mvnsite 2m 25s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 4m 42s the patch passed
        +1 javadoc 0m 27s hadoop-yarn-api in the patch passed.
        +1 javadoc 0m 53s hadoop-yarn-common in the patch passed.
        +1 javadoc 0m 30s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105)
        +1 javadoc 0m 18s hadoop-yarn-site in the patch passed.
              Other Tests
        +1 unit 0m 40s hadoop-yarn-api in the patch passed.
        +1 unit 2m 54s hadoop-yarn-common in the patch passed.
        -1 unit 14m 1s hadoop-yarn-server-nodemanager in the patch failed.
        +1 unit 0m 17s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 31s The patch does not generate ASF License warnings.
        79m 24s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881608/YARN-6930.003.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 8f2862adf020 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 8b242f0
        Default Java 1.8.0_144
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16873/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16873/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16873/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16873/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 24s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 16m 5s trunk passed +1 compile 9m 20s trunk passed +1 checkstyle 1m 0s trunk passed +1 mvnsite 2m 12s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 53s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 47s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 35s the patch passed +1 compile 6m 8s the patch passed +1 javac 6m 8s the patch passed +1 checkstyle 1m 4s the patch passed +1 mvnsite 2m 25s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 4m 42s the patch passed +1 javadoc 0m 27s hadoop-yarn-api in the patch passed. +1 javadoc 0m 53s hadoop-yarn-common in the patch passed. +1 javadoc 0m 30s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105) +1 javadoc 0m 18s hadoop-yarn-site in the patch passed.       Other Tests +1 unit 0m 40s hadoop-yarn-api in the patch passed. +1 unit 2m 54s hadoop-yarn-common in the patch passed. -1 unit 14m 1s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 0m 17s hadoop-yarn-site in the patch passed. +1 asflicense 0m 31s The patch does not generate ASF License warnings. 79m 24s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881608/YARN-6930.003.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8f2862adf020 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8b242f0 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16873/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16873/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16873/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16873/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 43s Maven dependency ordering for branch
        +1 mvninstall 13m 40s trunk passed
        +1 compile 9m 4s trunk passed
        +1 checkstyle 0m 49s trunk passed
        +1 mvnsite 2m 4s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
        +1 javadoc 1m 32s trunk passed
              Patch Compile Tests
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 23s the patch passed
        +1 compile 5m 35s the patch passed
        +1 javac 5m 35s the patch passed
        -0 checkstyle 0m 54s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 214 unchanged - 0 fixed = 215 total (was 214)
        +1 mvnsite 2m 2s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 3m 19s the patch passed
        +1 javadoc 0m 25s hadoop-yarn-api in the patch passed.
        +1 javadoc 0m 42s hadoop-yarn-common in the patch passed.
        +1 javadoc 0m 23s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105)
        +1 javadoc 0m 17s hadoop-yarn-site in the patch passed.
              Other Tests
        +1 unit 0m 33s hadoop-yarn-api in the patch passed.
        +1 unit 2m 42s hadoop-yarn-common in the patch passed.
        -1 unit 13m 59s hadoop-yarn-server-nodemanager in the patch failed.
        +1 unit 0m 12s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 22s The patch does not generate ASF License warnings.
        72m 35s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881289/YARN-6930.002.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux fe3e2f63f22a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 312e57b
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16830/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16830/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16830/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16830/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16830/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 13m 40s trunk passed +1 compile 9m 4s trunk passed +1 checkstyle 0m 49s trunk passed +1 mvnsite 2m 4s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 32s trunk passed       Patch Compile Tests 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 23s the patch passed +1 compile 5m 35s the patch passed +1 javac 5m 35s the patch passed -0 checkstyle 0m 54s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 214 unchanged - 0 fixed = 215 total (was 214) +1 mvnsite 2m 2s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 3m 19s the patch passed +1 javadoc 0m 25s hadoop-yarn-api in the patch passed. +1 javadoc 0m 42s hadoop-yarn-common in the patch passed. +1 javadoc 0m 23s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 104 unchanged - 1 fixed = 104 total (was 105) +1 javadoc 0m 17s hadoop-yarn-site in the patch passed.       Other Tests +1 unit 0m 33s hadoop-yarn-api in the patch passed. +1 unit 2m 42s hadoop-yarn-common in the patch passed. -1 unit 13m 59s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 0m 12s hadoop-yarn-site in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 72m 35s Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881289/YARN-6930.002.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux fe3e2f63f22a 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 312e57b Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16830/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16830/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16830/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16830/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16830/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        ebadger Eric Badger added a comment -

        Hey Shane Kumpf, thanks for the patch! I have a few comments

        If we're only allowing certain runtimes to be executed, then shouldn't we not even init them in DelegatingLinuxContainerRuntime.java:initialize()?

        I think DEFAULT_LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES and allowedRuntimes should be arrays of Strings. That way we can test equality against the array elements instead of whether a long string contains a substring.

        Should we set runtime = null if the runtime isn't allowed, just in case someone catches the ContainerExecutionException somewhere up the line?

        Show
        ebadger Eric Badger added a comment - Hey Shane Kumpf , thanks for the patch! I have a few comments If we're only allowing certain runtimes to be executed, then shouldn't we not even init them in DelegatingLinuxContainerRuntime.java:initialize() ? I think DEFAULT_LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES and allowedRuntimes should be arrays of Strings. That way we can test equality against the array elements instead of whether a long string contains a substring. Should we set runtime = null if the runtime isn't allowed, just in case someone catches the ContainerExecutionException somewhere up the line?
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Thank you Shane Kumpf. Yes, let's address the user issue in a different jira. Would you like to open it?

        94	  @Test
        95	  public void testIsRuntimeAllowedNotDocker() throws Exception {
        96	    conf.set(LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES,
        97	        "DefaultLinuxContainerRuntime,JavaSandboxLinuxContainerRuntime");
        98	    DelegatingLinuxContainerRuntime delegatingLinuxContainerRuntime =
        99	        new DelegatingLinuxContainerRuntime();
        100	    delegatingLinuxContainerRuntime.initialize(conf);
        101	    assertFalse(delegatingLinuxContainerRuntime.isRuntimeAllowed(
        102	        dockerLinuxContainerRuntime));
        103	  }
        

        This is nice, however I meant changing testIsRuntimeAllowedDefault to test that the default does not allow any of the other runtimes. This is what exactly checks what this jira is about.
        Also, could you add the new config into yarn-default.xml?

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you Shane Kumpf . Yes, let's address the user issue in a different jira. Would you like to open it? 94 @Test 95 public void testIsRuntimeAllowedNotDocker() throws Exception { 96 conf.set(LINUX_CONTAINER_RUNTIME_ALLOWED_RUNTIMES, 97 "DefaultLinuxContainerRuntime,JavaSandboxLinuxContainerRuntime" ); 98 DelegatingLinuxContainerRuntime delegatingLinuxContainerRuntime = 99 new DelegatingLinuxContainerRuntime(); 100 delegatingLinuxContainerRuntime.initialize(conf); 101 assertFalse(delegatingLinuxContainerRuntime.isRuntimeAllowed( 102 dockerLinuxContainerRuntime)); 103 } This is nice, however I meant changing testIsRuntimeAllowedDefault to test that the default does not allow any of the other runtimes. This is what exactly checks what this jira is about. Also, could you add the new config into yarn-default.xml?
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks for the comments, Miklos Szegedi. I will get those assertions added.

        I am wondering whether it would be a good idea to specify the user as well not just enabling a runtime in general. I could imagine that an admin allows Docker runtime only for specific users first...

        I think there could be value in an ACL model for container runtimes, however, there are additional issues regarding user squashing that need to be addressed before the feature is very useful. Let's address that in a different issue if that works for you?

        I'll also note that there are differences in implementation between the Java Sandbox and Docker runtimes, so I have not changed the behavior of the existing runtime selection with this patch. One of the considerations when developing the docker runtime was the ability to control the runtime per container as opposed to per application/cluster; i.e. AM's run as regular process based containers, while map and reduce task containers run using the docker runtime. The java sandbox based runtime takes a different approach and is enabled through configuration. This may be appropriate for the java sandbox runtime, but I don't want to change the way the docker runtime selection works today, which is why I decided not to introduce the docker-mode config and instead check the allowed runtimes after selection.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the comments, Miklos Szegedi . I will get those assertions added. I am wondering whether it would be a good idea to specify the user as well not just enabling a runtime in general. I could imagine that an admin allows Docker runtime only for specific users first... I think there could be value in an ACL model for container runtimes, however, there are additional issues regarding user squashing that need to be addressed before the feature is very useful. Let's address that in a different issue if that works for you? I'll also note that there are differences in implementation between the Java Sandbox and Docker runtimes, so I have not changed the behavior of the existing runtime selection with this patch. One of the considerations when developing the docker runtime was the ability to control the runtime per container as opposed to per application/cluster; i.e. AM's run as regular process based containers, while map and reduce task containers run using the docker runtime. The java sandbox based runtime takes a different approach and is enabled through configuration. This may be appropriate for the java sandbox runtime, but I don't want to change the way the docker runtime selection works today, which is why I decided not to introduce the docker-mode config and instead check the allowed runtimes after selection.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 17s Docker mode activated.
              Prechecks
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
              trunk Compile Tests
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 14m 43s trunk passed
        +1 compile 9m 31s trunk passed
        +1 checkstyle 1m 1s trunk passed
        +1 mvnsite 1m 28s trunk passed
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        -1 findbugs 0m 54s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings.
        +1 javadoc 1m 8s trunk passed
              Patch Compile Tests
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 33s the patch passed
        +1 compile 5m 51s the patch passed
        +1 javac 5m 51s the patch passed
        -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 215 unchanged - 0 fixed = 217 total (was 215)
        +1 mvnsite 1m 28s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
        +1 findbugs 2m 18s the patch passed
        -1 javadoc 0m 23s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 105 unchanged - 0 fixed = 106 total (was 105)
              Other Tests
        -1 unit 0m 33s hadoop-yarn-api in the patch failed.
        +1 unit 13m 26s hadoop-yarn-server-nodemanager in the patch passed.
        +1 unit 0m 14s hadoop-yarn-site in the patch passed.
        +1 asflicense 0m 26s The patch does not generate ASF License warnings.
        66m 29s



        Reason Tests
        Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:14b5c93
        JIRA Issue YARN-6930
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880714/YARN-6930.001.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 50390be84761 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / adb84f3
        Default Java 1.8.0_131
        findbugs v3.1.0-RC1
        findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16753/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16753/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16753/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/16753/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16753/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/16753/console
        Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.       trunk Compile Tests 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 14m 43s trunk passed +1 compile 9m 31s trunk passed +1 checkstyle 1m 1s trunk passed +1 mvnsite 1m 28s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 54s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 5 extant Findbugs warnings. +1 javadoc 1m 8s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 33s the patch passed +1 compile 5m 51s the patch passed +1 javac 5m 51s the patch passed -0 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 215 unchanged - 0 fixed = 217 total (was 215) +1 mvnsite 1m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 2m 18s the patch passed -1 javadoc 0m 23s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 1 new + 105 unchanged - 0 fixed = 106 total (was 105)       Other Tests -1 unit 0m 33s hadoop-yarn-api in the patch failed. +1 unit 13m 26s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 14s hadoop-yarn-site in the patch passed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 66m 29s Reason Tests Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6930 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12880714/YARN-6930.001.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 50390be84761 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / adb84f3 Default Java 1.8.0_131 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16753/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16753/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/16753/artifact/patchprocess/diff-javadoc-javadoc-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16753/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16753/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16753/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Thank you for the patch, Shane Kumpf. Would you mind adding two assertFalses verifying that Docker and Sandboxk are not allowed in . testIsRuntimeAllowedDefault?
        I am wondering whether it would be a good idea to specify the user as well not just enabling a runtime in general. I could imagine that an admin allows Docker runtime only for specific users first...

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you for the patch, Shane Kumpf . Would you mind adding two assertFalses verifying that Docker and Sandboxk are not allowed in . testIsRuntimeAllowedDefault ? I am wondering whether it would be a good idea to specify the user as well not just enabling a runtime in general. I could imagine that an admin allows Docker runtime only for specific users first...
        Hide
        gphillips Greg Phillips added a comment -

        Whitelisting runtimes seems to be the best option. I would likely modify the way sandbox-mode is selected to rely on the runtime whitelist and the container environment instead of using yarn.nodemanager.runtime.linux.sandbox-mode. This would remove the redundant knob issue.

        Show
        gphillips Greg Phillips added a comment - Whitelisting runtimes seems to be the best option. I would likely modify the way sandbox-mode is selected to rely on the runtime whitelist and the container environment instead of using yarn.nodemanager.runtime.linux.sandbox-mode . This would remove the redundant knob issue.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        + Greg Phillips, in case he would like to comment.

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - + Greg Phillips , in case he would like to comment.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        Can you use yarn.nodemanager.runtime.linux.sandbox-mode and yarn.nodemanager.runtime.linux.docker-mode for the global whitelist? I guess yarn.nodemanager.runtime.linux.sandbox-mode can also be renamed before beta1, if needed.

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Can you use yarn.nodemanager.runtime.linux.sandbox-mode and yarn.nodemanager.runtime.linux.docker-mode for the global whitelist? I guess yarn.nodemanager.runtime.linux.sandbox-mode can also be renamed before beta1, if needed.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks Miklos Szegedi - I did notice how it is being used after my comment. With the approach taken by JavaSandboxLinuxContainerRuntime, each runtime is now responsible for providing configuration knobs to disable that runtime. That means if the author of a new runtime forgets, there is no way to explicitly disable that runtime. It seems a global runtime whitelist would eliminate that concern, but having two configuration knobs isn't ideal either. What is your preference?

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks Miklos Szegedi - I did notice how it is being used after my comment. With the approach taken by JavaSandboxLinuxContainerRuntime, each runtime is now responsible for providing configuration knobs to disable that runtime. That means if the author of a new runtime forgets, there is no way to explicitly disable that runtime. It seems a global runtime whitelist would eliminate that concern, but having two configuration knobs isn't ideal either. What is your preference?
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - - edited

        The disable config is actually smart enough to set the mode in one configuration, when it is enabled. Are you saying that a user will need to enable sandbox with two different config knobs just because of this? There is no way to use the runtime, when it is disabled. See pickContainerRuntime()

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - - edited The disable config is actually smart enough to set the mode in one configuration, when it is enabled. Are you saying that a user will need to enable sandbox with two different config knobs just because of this? There is no way to use the runtime, when it is disabled. See pickContainerRuntime()
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        Thanks Miklos Szegedi for pointing that out. It seems that sandbox-mode is a bit different than what is being asked for in this issue. The sandbox-mode configuration is used to set the mode that is used within the policy (enforcing, permissive, or disabled - with disabled being the default) vs disabling the runtime all together.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - Thanks Miklos Szegedi for pointing that out. It seems that sandbox-mode is a bit different than what is being asked for in this issue. The sandbox-mode configuration is used to set the mode that is used within the policy (enforcing, permissive, or disabled - with disabled being the default) vs disabling the runtime all together.
        Hide
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

        JavaSandboxLinuxContainerRuntime already works like this. See yarn.nodemanager.runtime.linux.sandbox-mode for details. It is disabled by default. I suggest updating the title to DockerLinuxContainerRuntime only.

        Show
        miklos.szegedi@cloudera.com Miklos Szegedi added a comment - JavaSandboxLinuxContainerRuntime already works like this. See yarn.nodemanager.runtime.linux.sandbox-mode for details. It is disabled by default. I suggest updating the title to DockerLinuxContainerRuntime only.
        Hide
        shanekumpf@gmail.com Shane Kumpf added a comment -

        I can take this one.

        Show
        shanekumpf@gmail.com Shane Kumpf added a comment - I can take this one.

          People

          • Assignee:
            shanekumpf@gmail.com Shane Kumpf
            Reporter:
            vinodkv Vinod Kumar Vavilapalli
          • Votes:
            0 Vote for this issue
            Watchers:
            13 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development