Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5280

Allow YARN containers to run with Java Security Manager

    Details

    • Target Version/s:
    • Hadoop Flags:
      Reviewed
    • Flags:
      Patch

      Description

      YARN applications have the ability to perform privileged actions which have the potential to add instability into the cluster. The Java Security Manager can be used to prevent users from running privileged actions while still allowing their core data processing use cases.
      Introduce a YARN flag which will allow a Hadoop administrator to enable the Java Security Manager for user code, while still providing complete permissions to core Hadoop libraries.

      1. YARN-5280.001.patch
        39 kB
        Greg Phillips
      2. YARN-5280.002.patch
        39 kB
        Greg Phillips
      3. YARN-5280.003.patch
        40 kB
        Greg Phillips
      4. YARN-5280.004.patch
        43 kB
        Greg Phillips
      5. YARN-5280.005.patch
        57 kB
        Greg Phillips
      6. YARN-5280.006.patch
        60 kB
        Greg Phillips
      7. YARN-5280.007.patch
        62 kB
        Greg Phillips
      8. YARN-5280.008.patch
        63 kB
        Greg Phillips
      9. YARN-5280.patch
        17 kB
        Greg Phillips
      10. YARNContainerSandbox.pdf
        188 kB
        Greg Phillips

        Activity

        Hide
        hudson Hudson added a comment -

        SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11328 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11328/)
        YARN-5280. Allow YARN containers to run with Java Security Manager (rkanter: rev 6f6dfe0202249c129b36edfd145a2224140139cc)

        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/LinuxContainerRuntimeConstants.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/JavaSandboxLinuxContainerRuntime.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestJavaSandboxLinuxContainerRuntime.java
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DelegatingLinuxContainerRuntime.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/executor/ContainerPrepareContext.java
        • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/java.policy
        • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
        Show
        hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #11328 (See https://builds.apache.org/job/Hadoop-trunk-Commit/11328/ ) YARN-5280 . Allow YARN containers to run with Java Security Manager (rkanter: rev 6f6dfe0202249c129b36edfd145a2224140139cc) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/LinuxContainerExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/LinuxContainerRuntimeConstants.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/JavaSandboxLinuxContainerRuntime.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestJavaSandboxLinuxContainerRuntime.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DelegatingLinuxContainerRuntime.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/executor/ContainerPrepareContext.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/resources/java.policy (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
        Hide
        rkanter Robert Kanter added a comment -

        Thanks Greg Phillips and others for reviews. Committed to trunk!

        Show
        rkanter Robert Kanter added a comment - Thanks Greg Phillips and others for reviews. Committed to trunk!
        Hide
        rkanter Robert Kanter added a comment -

        Greg Phillips, you're right: that was TestContainerManagerSecurity being flaky.

        +1

        Will commit soon

        Show
        rkanter Robert Kanter added a comment - Greg Phillips , you're right: that was TestContainerManagerSecurity being flaky. +1 Will commit soon
        Hide
        vvasudev Varun Vasudev added a comment -

        Ah I see. Thanks for the clarification!

        Show
        vvasudev Varun Vasudev added a comment - Ah I see. Thanks for the clarification!
        Hide
        gphillips Greg Phillips added a comment - - edited

        Varun Vasudev Thanks for reviewing the patch. The ContainerRuntimeContext is used across all methods in the ContainerRuntime interface:

        ContainerRuntime.java
          void prepareContainer(ContainerRuntimeContext ctx)
              throws ContainerExecutionException;
          void launchContainer(ContainerRuntimeContext ctx)
              throws ContainerExecutionException;
          void signalContainer(ContainerRuntimeContext ctx)
              throws ContainerExecutionException;
          void reapContainer(ContainerRuntimeContext ctx)
              throws ContainerExecutionException;
        

        The goal was to conform to the existing ContainerRuntime interface, though it definitely could make sense to merge the various Context's in a separate ticket.

        Show
        gphillips Greg Phillips added a comment - - edited Varun Vasudev Thanks for reviewing the patch. The ContainerRuntimeContext is used across all methods in the ContainerRuntime interface: ContainerRuntime.java void prepareContainer(ContainerRuntimeContext ctx) throws ContainerExecutionException; void launchContainer(ContainerRuntimeContext ctx) throws ContainerExecutionException; void signalContainer(ContainerRuntimeContext ctx) throws ContainerExecutionException; void reapContainer(ContainerRuntimeContext ctx) throws ContainerExecutionException; The goal was to conform to the existing ContainerRuntime interface, though it definitely could make sense to merge the various Context's in a separate ticket.
        Hide
        vvasudev Varun Vasudev added a comment - - edited

        Greg Phillips - can you explain this -

        +  public void prepareContainer(ContainerPrepareContext ctx) throws IOException {
        +
        +    ContainerRuntimeContext.Builder builder =
        +        new ContainerRuntimeContext.Builder(ctx.getContainer());
        

        Shouldn't it be ContainerPrepareContext.Builder?

        Similarly,

         +  @Override
        +  public void prepareContainer(ContainerRuntimeContext ctx)
        +      throws ContainerExecutionException {
        

        should use the ContainerPrepareContext no?

        Show
        vvasudev Varun Vasudev added a comment - - edited Greg Phillips - can you explain this - + public void prepareContainer(ContainerPrepareContext ctx) throws IOException { + + ContainerRuntimeContext.Builder builder = + new ContainerRuntimeContext.Builder(ctx.getContainer()); Shouldn't it be ContainerPrepareContext.Builder? Similarly, + @Override + public void prepareContainer(ContainerRuntimeContext ctx) + throws ContainerExecutionException { should use the ContainerPrepareContext no?
        Hide
        gphillips Greg Phillips added a comment -

        Robert Kanter I tested TestContainerManagerSecurity with and without 008 and they each failed ~70% of the time (across ~20 runs each). There seems to be a race condition introduced in YARN-2584.

        TestContainerManagerSecurity.java
            while ((interval-- > 0)
                && !nmContet.getContainers().get(containerId)
                  .cloneAndGetContainerStatus().getState()
                  .equals(ContainerState.COMPLETE)) {
        

        The nmContet.getContainers().get(containerId) can return null. It seems the race is between the container being set to complete and it being completely removed from the Map. In any case it seems to be unrelated to YARN-5280, I will open another ticket to address this issue if you agree.

        Show
        gphillips Greg Phillips added a comment - Robert Kanter I tested TestContainerManagerSecurity with and without 008 and they each failed ~70% of the time (across ~20 runs each). There seems to be a race condition introduced in YARN-2584 . TestContainerManagerSecurity.java while ((interval-- > 0) && !nmContet.getContainers().get(containerId) .cloneAndGetContainerStatus().getState() .equals(ContainerState.COMPLETE)) { The nmContet.getContainers().get(containerId) can return null. It seems the race is between the container being set to complete and it being completely removed from the Map. In any case it seems to be unrelated to YARN-5280 , I will open another ticket to address this issue if you agree.
        Hide
        rkanter Robert Kanter added a comment -

        Greg Phillips, can you take a look at TestContainerManagerSecurity? It seems to fail now with the 008 patch applied. Something must have changed in the last week.

        -------------------------------------------------------
         T E S T S
        -------------------------------------------------------
        Picked up _JAVA_OPTIONS: -Djava.awt.headless=true
        Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity
        Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 41.505 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity
        testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity)  Time elapsed: 17.237 sec  <<< ERROR!
        java.lang.NullPointerException: null
        	at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:399)
        	at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:342)
        	at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:159)
        
        
        Results :
        
        Tests in error:
          TestContainerManagerSecurity.testContainerManager:159->testNMTokens:342->waitForContainerToFinishOnNM:399 NullPointer
        
        Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
        
        Show
        rkanter Robert Kanter added a comment - Greg Phillips , can you take a look at TestContainerManagerSecurity ? It seems to fail now with the 008 patch applied. Something must have changed in the last week. ------------------------------------------------------- T E S T S ------------------------------------------------------- Picked up _JAVA_OPTIONS: -Djava.awt.headless=true Running org.apache.hadoop.yarn.server.TestContainerManagerSecurity Tests run: 2, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 41.505 sec <<< FAILURE! - in org.apache.hadoop.yarn.server.TestContainerManagerSecurity testContainerManager[1](org.apache.hadoop.yarn.server.TestContainerManagerSecurity) Time elapsed: 17.237 sec <<< ERROR! java.lang.NullPointerException: null at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.waitForContainerToFinishOnNM(TestContainerManagerSecurity.java:399) at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testNMTokens(TestContainerManagerSecurity.java:342) at org.apache.hadoop.yarn.server.TestContainerManagerSecurity.testContainerManager(TestContainerManagerSecurity.java:159) Results : Tests in error: TestContainerManagerSecurity.testContainerManager:159->testNMTokens:342->waitForContainerToFinishOnNM:399 NullPointer Tests run: 2, Failures: 0, Errors: 1, Skipped: 0
        Hide
        rkanter Robert Kanter added a comment -

        Will commit this later today if no other comments

        Show
        rkanter Robert Kanter added a comment - Will commit this later today if no other comments
        Hide
        rkanter Robert Kanter added a comment -

        From your previous note it became clear simply limiting the container to execute something called 'java' is inherently insecure.

        Yup

        This patch ensures the java executable used to run the NodeManager is also used for the container. If a different version of Java or even a shell script named 'java' is provided an exception will be thrown in enforcing mode, or a warning will be logged in permissive mode.

        That's a good idea.

        The 008 patch LGTM +1

        Larry McCay, Varun Vasudev any other comments?

        Show
        rkanter Robert Kanter added a comment - From your previous note it became clear simply limiting the container to execute something called 'java' is inherently insecure. Yup This patch ensures the java executable used to run the NodeManager is also used for the container. If a different version of Java or even a shell script named 'java' is provided an exception will be thrown in enforcing mode, or a warning will be logged in permissive mode. That's a good idea. The 008 patch LGTM +1 Larry McCay , Varun Vasudev any other comments?
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 16s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 14m 4s trunk passed
        +1 compile 7m 8s trunk passed
        +1 checkstyle 0m 47s trunk passed
        +1 mvnsite 1m 40s trunk passed
        +1 mvneclipse 0m 55s trunk passed
        +1 findbugs 3m 2s trunk passed
        +1 javadoc 1m 22s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 17s the patch passed
        +1 compile 6m 8s the patch passed
        +1 javac 6m 8s the patch passed
        -0 checkstyle 0m 47s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 268 unchanged - 2 fixed = 273 total (was 270)
        +1 mvnsite 1m 37s the patch passed
        +1 mvneclipse 0m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 21s the patch passed
        +1 javadoc 1m 17s the patch passed
        +1 unit 0m 31s hadoop-yarn-api in the patch passed.
        +1 unit 2m 30s hadoop-yarn-common in the patch passed.
        +1 unit 13m 58s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        71m 11s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5280
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852664/YARN-5280.008.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 5391d5466fbc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aaf2713
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14942/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14942/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14942/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 14m 4s trunk passed +1 compile 7m 8s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 1m 40s trunk passed +1 mvneclipse 0m 55s trunk passed +1 findbugs 3m 2s trunk passed +1 javadoc 1m 22s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 6m 8s the patch passed +1 javac 6m 8s the patch passed -0 checkstyle 0m 47s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 268 unchanged - 2 fixed = 273 total (was 270) +1 mvnsite 1m 37s the patch passed +1 mvneclipse 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 21s the patch passed +1 javadoc 1m 17s the patch passed +1 unit 0m 31s hadoop-yarn-api in the patch passed. +1 unit 2m 30s hadoop-yarn-common in the patch passed. +1 unit 13m 58s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 71m 11s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5280 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852664/YARN-5280.008.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 5391d5466fbc 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aaf2713 Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14942/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14942/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14942/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        gphillips Greg Phillips added a comment -

        Robert Kanter - From your previous note it became clear simply limiting the container to execute something called 'java' is inherently insecure. This patch ensures the java executable used to run the NodeManager is also used for the container. If a different version of Java or even a shell script named 'java' is provided an exception will be thrown in enforcing mode, or a warning will be logged in permissive mode.

        Show
        gphillips Greg Phillips added a comment - Robert Kanter - From your previous note it became clear simply limiting the container to execute something called 'java' is inherently insecure. This patch ensures the java executable used to run the NodeManager is also used for the container. If a different version of Java or even a shell script named 'java' is provided an exception will be thrown in enforcing mode, or a warning will be logged in permissive mode.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 15s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 12m 36s trunk passed
        +1 compile 8m 33s trunk passed
        +1 checkstyle 0m 48s trunk passed
        +1 mvnsite 1m 37s trunk passed
        +1 mvneclipse 0m 56s trunk passed
        +1 findbugs 2m 54s trunk passed
        +1 javadoc 1m 20s trunk passed
        0 mvndep 0m 9s Maven dependency ordering for patch
        +1 mvninstall 1m 13s the patch passed
        +1 compile 6m 43s the patch passed
        +1 javac 6m 43s the patch passed
        -0 checkstyle 0m 46s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 268 unchanged - 2 fixed = 273 total (was 270)
        +1 mvnsite 1m 33s the patch passed
        +1 mvneclipse 0m 53s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 17s the patch passed
        +1 javadoc 1m 17s the patch passed
        +1 unit 0m 32s hadoop-yarn-api in the patch passed.
        +1 unit 2m 26s hadoop-yarn-common in the patch passed.
        +1 unit 13m 58s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        71m 30s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5280
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852580/YARN-5280.007.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux 8817ea5bd916 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / aaf106f
        Default Java 1.8.0_121
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14931/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14931/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14931/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 15s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 12m 36s trunk passed +1 compile 8m 33s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 1m 37s trunk passed +1 mvneclipse 0m 56s trunk passed +1 findbugs 2m 54s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 6m 43s the patch passed +1 javac 6m 43s the patch passed -0 checkstyle 0m 46s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 268 unchanged - 2 fixed = 273 total (was 270) +1 mvnsite 1m 33s the patch passed +1 mvneclipse 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 17s the patch passed +1 javadoc 1m 17s the patch passed +1 unit 0m 32s hadoop-yarn-api in the patch passed. +1 unit 2m 26s hadoop-yarn-common in the patch passed. +1 unit 13m 58s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 71m 30s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5280 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12852580/YARN-5280.007.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 8817ea5bd916 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2 15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / aaf106f Default Java 1.8.0_121 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14931/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14931/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14931/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        gphillips Greg Phillips added a comment -

        Robert Kanter - Thanks for your insights. I have added tests for every regex which demonstrate positive/negative cases. Additionally the patterns have been modified to ensure the passed command will instantiate a JVM.

        Application submitters are still able to provide arbitrary commands to launch the ApplicationMaster. If they provide command which doesn't use java it will be blocked in 'enforcing' mode. Otherwise they could potentially call any version of java which exists on the system where the Application Master is allocated (including /evil/java).

        Show
        gphillips Greg Phillips added a comment - Robert Kanter - Thanks for your insights. I have added tests for every regex which demonstrate positive/negative cases. Additionally the patterns have been modified to ensure the passed command will instantiate a JVM. Application submitters are still able to provide arbitrary commands to launch the ApplicationMaster. If they provide command which doesn't use java it will be blocked in 'enforcing' mode. Otherwise they could potentially call any version of java which exists on the system where the Application Master is allocated (including /evil/java).
        Hide
        rkanter Robert Kanter added a comment - - edited

        The 006 patch looks mostly good to me. My only remaining concern is with the regexes in NMContainerPolicyUtils. It would be good to have some tests that specifically verify the regexes with some reasonable positive and negative cases.

        I'm also wondering if there's some way a malicious user could trick it (I might not be following the regexes correctly here so feel free to let me know if something like this can't actually happen). For example, if we're removing java, and you have /evil/jajavava, it's going to become /evil/java.

        Show
        rkanter Robert Kanter added a comment - - edited The 006 patch looks mostly good to me. My only remaining concern is with the regexes in NMContainerPolicyUtils . It would be good to have some tests that specifically verify the regexes with some reasonable positive and negative cases. I'm also wondering if there's some way a malicious user could trick it (I might not be following the regexes correctly here so feel free to let me know if something like this can't actually happen). For example, if we're removing java , and you have /evil/jajavava , it's going to become /evil/java .
        Hide
        gphillips Greg Phillips added a comment -

        Varun Vasudev - I encountered some issues when attempting to move the generated java.policy files into the container or application directories due to permissions conflicts when running in secure mode. Namely there are no container or application specific directories which allow write access for the yarn user, and read access to the container run-as user in all configurations. This is resolved using the hadoop.tmp.dir following the example set by the DockerRuntime. The risk of running out of space on hadoop.tmp.dir should be small due to the following:

        1. Generated policy files are ~4KB, the largest yarn nodes can handle around 500 containers making the hypothetical upper bound ~2MB of tmp usage.
        2. Policy files are deleted at the completion of container launch regardless of exit value, as well as on nodemanager restart. This functionality has been moved from reapContainer to the end of launchContainer.

        Once we have the runtime support in, we can add support in MR and distributed shell for the feature.

        This patch has been tested extensively with MR to ensure all components (distributed cache, libjars, etc.) work as intended. The distributed shell will work if the distributed shell jar is available under the hadoop home directory since all libraries in the hadoop home directory are granted all permissions. Cluster administrators will likely want to limit access to the distributed shell jar to harden the cluster.

        Please let me know if these compromises seem appropriate, or if there are additional steps required to make this feature viable.

        Show
        gphillips Greg Phillips added a comment - Varun Vasudev - I encountered some issues when attempting to move the generated java.policy files into the container or application directories due to permissions conflicts when running in secure mode. Namely there are no container or application specific directories which allow write access for the yarn user, and read access to the container run-as user in all configurations. This is resolved using the hadoop.tmp.dir following the example set by the DockerRuntime. The risk of running out of space on hadoop.tmp.dir should be small due to the following: Generated policy files are ~4KB, the largest yarn nodes can handle around 500 containers making the hypothetical upper bound ~2MB of tmp usage. Policy files are deleted at the completion of container launch regardless of exit value, as well as on nodemanager restart. This functionality has been moved from reapContainer to the end of launchContainer. Once we have the runtime support in, we can add support in MR and distributed shell for the feature. This patch has been tested extensively with MR to ensure all components (distributed cache, libjars, etc.) work as intended. The distributed shell will work if the distributed shell jar is available under the hadoop home directory since all libraries in the hadoop home directory are granted all permissions. Cluster administrators will likely want to limit access to the distributed shell jar to harden the cluster. Please let me know if these compromises seem appropriate, or if there are additional steps required to make this feature viable.
        Hide
        hadoopqa Hadoop QA added a comment -
        +1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 12s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 12m 41s trunk passed
        +1 compile 4m 54s trunk passed
        +1 checkstyle 0m 46s trunk passed
        +1 mvnsite 1m 36s trunk passed
        +1 mvneclipse 0m 55s trunk passed
        +1 findbugs 2m 59s trunk passed
        +1 javadoc 1m 20s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 14s the patch passed
        +1 compile 4m 38s the patch passed
        +1 javac 4m 38s the patch passed
        -0 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 267 unchanged - 2 fixed = 272 total (was 269)
        +1 mvnsite 1m 34s the patch passed
        +1 mvneclipse 0m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 2s The patch has no ill-formed XML file.
        +1 findbugs 3m 22s the patch passed
        +1 javadoc 1m 17s the patch passed
        +1 unit 0m 31s hadoop-yarn-api in the patch passed.
        +1 unit 2m 23s hadoop-yarn-common in the patch passed.
        +1 unit 13m 0s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 29s The patch does not generate ASF License warnings.
        64m 2s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5280
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845844/YARN-5280.006.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux dcaeed50ee29 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 0a55bd8
        Default Java 1.8.0_111
        findbugs v3.0.0
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14571/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14571/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14571/console
        Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - +1 overall Vote Subsystem Runtime Comment 0 reexec 0m 12s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 12m 41s trunk passed +1 compile 4m 54s trunk passed +1 checkstyle 0m 46s trunk passed +1 mvnsite 1m 36s trunk passed +1 mvneclipse 0m 55s trunk passed +1 findbugs 2m 59s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 14s the patch passed +1 compile 4m 38s the patch passed +1 javac 4m 38s the patch passed -0 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn: The patch generated 5 new + 267 unchanged - 2 fixed = 272 total (was 269) +1 mvnsite 1m 34s the patch passed +1 mvneclipse 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. +1 findbugs 3m 22s the patch passed +1 javadoc 1m 17s the patch passed +1 unit 0m 31s hadoop-yarn-api in the patch passed. +1 unit 2m 23s hadoop-yarn-common in the patch passed. +1 unit 13m 0s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 64m 2s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5280 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12845844/YARN-5280.006.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux dcaeed50ee29 3.13.0-103-generic #150-Ubuntu SMP Thu Nov 24 10:34:17 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 0a55bd8 Default Java 1.8.0_111 findbugs v3.0.0 checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14571/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14571/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14571/console Powered by Apache Yetus 0.5.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        gphillips Greg Phillips added a comment -

        This patch reverts to writing the java policy files into hadoop.tmp.dir following the example set by the DockerContainerRuntime. This is necessary since yarn local directories will not allow yarn to write the file and the container run-as user to access the file in secure configurations. This patch ensures any generated policy files will be deleted to mitigate the possibility of hadoop.tmp.dir running out of space.

        Usages of ContainerRuntime#reapContainer have been removed from the patch as recommended.

        Show
        gphillips Greg Phillips added a comment - This patch reverts to writing the java policy files into hadoop.tmp.dir following the example set by the DockerContainerRuntime. This is necessary since yarn local directories will not allow yarn to write the file and the container run-as user to access the file in secure configurations. This patch ensures any generated policy files will be deleted to mitigate the possibility of hadoop.tmp.dir running out of space. Usages of ContainerRuntime#reapContainer have been removed from the patch as recommended.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 11s Maven dependency ordering for branch
        +1 mvninstall 8m 1s trunk passed
        +1 compile 5m 7s trunk passed
        +1 checkstyle 0m 47s trunk passed
        +1 mvnsite 1m 40s trunk passed
        +1 mvneclipse 0m 54s trunk passed
        +1 findbugs 2m 58s trunk passed
        +1 javadoc 1m 18s trunk passed
        0 mvndep 0m 11s Maven dependency ordering for patch
        +1 mvninstall 1m 13s the patch passed
        +1 compile 4m 50s the patch passed
        -1 javac 4m 50s hadoop-yarn-project_hadoop-yarn generated 10 new + 34 unchanged - 0 fixed = 44 total (was 34)
        -0 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn: The patch generated 15 new + 267 unchanged - 1 fixed = 282 total (was 268)
        +1 mvnsite 1m 39s the patch passed
        +1 mvneclipse 0m 51s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 21s the patch passed
        +1 javadoc 1m 20s the patch passed
        +1 unit 0m 32s hadoop-yarn-api in the patch passed.
        +1 unit 2m 26s hadoop-yarn-common in the patch passed.
        +1 unit 13m 46s hadoop-yarn-server-nodemanager in the patch passed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        61m 1s



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5280
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839885/YARN-5280.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux a545b98d7804 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 683e0c7
        Default Java 1.8.0_111
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/14000/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14000/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14000/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/14000/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 8m 1s trunk passed +1 compile 5m 7s trunk passed +1 checkstyle 0m 47s trunk passed +1 mvnsite 1m 40s trunk passed +1 mvneclipse 0m 54s trunk passed +1 findbugs 2m 58s trunk passed +1 javadoc 1m 18s trunk passed 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 1m 13s the patch passed +1 compile 4m 50s the patch passed -1 javac 4m 50s hadoop-yarn-project_hadoop-yarn generated 10 new + 34 unchanged - 0 fixed = 44 total (was 34) -0 checkstyle 0m 45s hadoop-yarn-project/hadoop-yarn: The patch generated 15 new + 267 unchanged - 1 fixed = 282 total (was 268) +1 mvnsite 1m 39s the patch passed +1 mvneclipse 0m 51s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 21s the patch passed +1 javadoc 1m 20s the patch passed +1 unit 0m 32s hadoop-yarn-api in the patch passed. +1 unit 2m 26s hadoop-yarn-common in the patch passed. +1 unit 13m 46s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 61m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5280 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839885/YARN-5280.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux a545b98d7804 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 683e0c7 Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/14000/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/14000/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/14000/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/14000/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 18s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 9s Maven dependency ordering for branch
        +1 mvninstall 6m 48s trunk passed
        +1 compile 4m 56s trunk passed
        +1 checkstyle 0m 46s trunk passed
        +1 mvnsite 1m 40s trunk passed
        +1 mvneclipse 0m 56s trunk passed
        +1 findbugs 2m 54s trunk passed
        +1 javadoc 1m 20s trunk passed
        0 mvndep 0m 10s Maven dependency ordering for patch
        +1 mvninstall 1m 12s the patch passed
        +1 compile 4m 35s the patch passed
        -1 javac 4m 35s hadoop-yarn-project_hadoop-yarn generated 10 new + 34 unchanged - 0 fixed = 44 total (was 34)
        -0 checkstyle 0m 46s hadoop-yarn-project/hadoop-yarn: The patch generated 15 new + 267 unchanged - 1 fixed = 282 total (was 268)
        +1 mvnsite 1m 38s the patch passed
        +1 mvneclipse 0m 53s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 xml 0m 1s The patch has no ill-formed XML file.
        +1 findbugs 3m 16s the patch passed
        +1 javadoc 1m 17s the patch passed
        +1 unit 0m 30s hadoop-yarn-api in the patch passed.
        +1 unit 2m 23s hadoop-yarn-common in the patch passed.
        -1 unit 13m 42s hadoop-yarn-server-nodemanager in the patch failed.
        +1 asflicense 0m 30s The patch does not generate ASF License warnings.
        59m 4s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestJavaSandboxLinuxContainerRuntime



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:a9ad5d6
        JIRA Issue YARN-5280
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839810/YARN-5280.005.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
        uname Linux e92e68afe528 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / f922067
        Default Java 1.8.0_111
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/13993/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13993/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13993/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13993/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13993/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 9s Maven dependency ordering for branch +1 mvninstall 6m 48s trunk passed +1 compile 4m 56s trunk passed +1 checkstyle 0m 46s trunk passed +1 mvnsite 1m 40s trunk passed +1 mvneclipse 0m 56s trunk passed +1 findbugs 2m 54s trunk passed +1 javadoc 1m 20s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 1m 12s the patch passed +1 compile 4m 35s the patch passed -1 javac 4m 35s hadoop-yarn-project_hadoop-yarn generated 10 new + 34 unchanged - 0 fixed = 44 total (was 34) -0 checkstyle 0m 46s hadoop-yarn-project/hadoop-yarn: The patch generated 15 new + 267 unchanged - 1 fixed = 282 total (was 268) +1 mvnsite 1m 38s the patch passed +1 mvneclipse 0m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 3m 16s the patch passed +1 javadoc 1m 17s the patch passed +1 unit 0m 30s hadoop-yarn-api in the patch passed. +1 unit 2m 23s hadoop-yarn-common in the patch passed. -1 unit 13m 42s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 30s The patch does not generate ASF License warnings. 59m 4s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestJavaSandboxLinuxContainerRuntime Subsystem Report/Notes Docker Image:yetus/hadoop:a9ad5d6 JIRA Issue YARN-5280 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12839810/YARN-5280.005.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux e92e68afe528 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / f922067 Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13993/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13993/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13993/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13993/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13993/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        gphillips Greg Phillips added a comment -

        Removed application queue dependency for whitelisting. Whitelisting now uses a user group to allow users to opt out of using the JVMContainerSandbox on a job by job basis.

        Generated java.policy files are now written to the application private directory. Users will not be able to access the policy file itself, but will be able to inspect the policy from within the container using System#getSecurityManager.

        Container preparation functionality has been moved from LinuxContainerExecutor#writeLaunchEnv to LinuxContainerExecutor#prepareContainer. ContainerExecutor#prepareContainer is called from ContainerLaunch#call prior to writeLaunchEnv. Additionally a ContainerPrepareContext has been created as the only parameter to ContainerExecutor#prepareContainer.

        Show
        gphillips Greg Phillips added a comment - Removed application queue dependency for whitelisting. Whitelisting now uses a user group to allow users to opt out of using the JVMContainerSandbox on a job by job basis. Generated java.policy files are now written to the application private directory. Users will not be able to access the policy file itself, but will be able to inspect the policy from within the container using System#getSecurityManager. Container preparation functionality has been moved from LinuxContainerExecutor#writeLaunchEnv to LinuxContainerExecutor#prepareContainer. ContainerExecutor#prepareContainer is called from ContainerLaunch#call prior to writeLaunchEnv. Additionally a ContainerPrepareContext has been created as the only parameter to ContainerExecutor#prepareContainer.
        Hide
        gphillips Greg Phillips added a comment -

        That sounds fantastic. I will add those changes to the next patch.

        Show
        gphillips Greg Phillips added a comment - That sounds fantastic. I will add those changes to the next patch.
        Hide
        vvasudev Varun Vasudev added a comment -

        The difficulty arises when moving the functionality from prepareContainer to launchContainer. In particular I need to modify the actual java run command instead of the container launch command. The only way I have found to modify the run command found within the launch_container.sh is through the LinuxContainerExecutor#writeLaunchEnv. A method which links the LinuxContainerExecutor with the ContainerRuntime prior to the environment being written seems necessary for this feature. I am very interested in your thoughts on this matter.

        Ah you're correct. I missed this. How about we add a new method called prepareContainer in the ContainerExecutor base class which does nothing by default and override it in the LinuxContainerExecutor class to call the runtime's prepareContainer method? We can call this method before we call writeLaunchEnv. That should solve your requirement, correct?

        Show
        vvasudev Varun Vasudev added a comment - The difficulty arises when moving the functionality from prepareContainer to launchContainer. In particular I need to modify the actual java run command instead of the container launch command. The only way I have found to modify the run command found within the launch_container.sh is through the LinuxContainerExecutor#writeLaunchEnv. A method which links the LinuxContainerExecutor with the ContainerRuntime prior to the environment being written seems necessary for this feature. I am very interested in your thoughts on this matter. Ah you're correct. I missed this. How about we add a new method called prepareContainer in the ContainerExecutor base class which does nothing by default and override it in the LinuxContainerExecutor class to call the runtime's prepareContainer method? We can call this method before we call writeLaunchEnv. That should solve your requirement, correct?
        Hide
        gphillips Greg Phillips added a comment -

        Varun Vasudev - thanks for the guidance. I have definitely run out of space in the hadoop tmp dir in the past, and I completely agree that storing the java.policy in the container private directory is a better solution. I have made that modification, and I am currently testing it. For debugging purposes users can inspect the generated java.policy file from within their application using System.getSecurityManager(), or by providing client arguments for security manager debugging. I will include notes on this in the javadoc, and in future feature documentation.

        The difficulty arises when moving the functionality from prepareContainer to launchContainer. In particular I need to modify the actual java run command instead of the container launch command. The only way I have found to modify the run command found within the launch_container.sh is through the LinuxContainerExecutor#writeLaunchEnv. A method which links the LinuxContainerExecutor with the ContainerRuntime prior to the environment being written seems necessary for this feature. I am very interested in your thoughts on this matter.

        Show
        gphillips Greg Phillips added a comment - Varun Vasudev - thanks for the guidance. I have definitely run out of space in the hadoop tmp dir in the past, and I completely agree that storing the java.policy in the container private directory is a better solution. I have made that modification, and I am currently testing it. For debugging purposes users can inspect the generated java.policy file from within their application using System.getSecurityManager(), or by providing client arguments for security manager debugging. I will include notes on this in the javadoc, and in future feature documentation. The difficulty arises when moving the functionality from prepareContainer to launchContainer. In particular I need to modify the actual java run command instead of the container launch command. The only way I have found to modify the run command found within the launch_container.sh is through the LinuxContainerExecutor#writeLaunchEnv. A method which links the LinuxContainerExecutor with the ContainerRuntime prior to the environment being written seems necessary for this feature. I am very interested in your thoughts on this matter.
        Hide
        vvasudev Varun Vasudev added a comment -

        Thanks for the explanation Sidharta Seethana. Greg Phillips - do you think you can move all the functionality of prepareContainer into launchContainer and then call super.launchContainer in JavaSandboxLinuxContainerRuntime? The benefits are -
        1) You won't be affected if/when we do get round to figuring out where to call prepareContainer.
        2) The environment will also be available to you as part of the ContainerStartContext.
        3) Modifying the launch command is more natural in launchContainer than prepareContainer

        A question about the policy file - do you think this is something that end users should be able to view to help debug applications?

        My suggestion is to not use the hadoop tmp dir for the policy file but instead use the container private directory. You can add the container private directory to the ContainerStartContext in ContainerLaunch#call and ContainerRelaunch#call. That way -
        1) You don't need to worry about the hadoop tmp dir running out of space(which we've seen in a few cases)
        2) The policy file will be cleaned up for you by YARN and you can get rid of the reapContainer functionality you have.
        3) You can also potentially re-use the same policy file across container restarts instead of creating a temporary file every time, since container private directories are only for the container.

        With regards to the patch you have which resolves the testing errors and removes the use of YARN queues - please include those changes in the next patch. Once we have the runtime support in, we can add support in MR and distributed shell for the feature.

        Show
        vvasudev Varun Vasudev added a comment - Thanks for the explanation Sidharta Seethana . Greg Phillips - do you think you can move all the functionality of prepareContainer into launchContainer and then call super.launchContainer in JavaSandboxLinuxContainerRuntime? The benefits are - 1) You won't be affected if/when we do get round to figuring out where to call prepareContainer. 2) The environment will also be available to you as part of the ContainerStartContext. 3) Modifying the launch command is more natural in launchContainer than prepareContainer A question about the policy file - do you think this is something that end users should be able to view to help debug applications? My suggestion is to not use the hadoop tmp dir for the policy file but instead use the container private directory. You can add the container private directory to the ContainerStartContext in ContainerLaunch#call and ContainerRelaunch#call. That way - 1) You don't need to worry about the hadoop tmp dir running out of space(which we've seen in a few cases) 2) The policy file will be cleaned up for you by YARN and you can get rid of the reapContainer functionality you have. 3) You can also potentially re-use the same policy file across container restarts instead of creating a temporary file every time, since container private directories are only for the container. With regards to the patch you have which resolves the testing errors and removes the use of YARN queues - please include those changes in the next patch. Once we have the runtime support in, we can add support in MR and distributed shell for the feature.
        Hide
        sidharta-s Sidharta Seethana added a comment -

        YARN-3853 added the ContainerRuntime interface. The objective of adding the ‘prepareContainer()’ and ‘reapContainer()’ methods was to provide finer grained control of the container lifecycle (and at some point add corresponding instrumentation to track where time is spent in the container lifecycle). Using docker containers as an example, ‘prepareContainer()’ could be hooked into docker ‘create’ (and maybe even image localization). reapContainer() could be used for (optional) post complete container deletion.

        Once container runtimes were introduced, the interaction with resource handlers got a bit … trickier. Right now, the same cgroups based resource handlers can be used across the ‘default’ and ‘docker’ container runtimes (mainly due to docker’s cgroup-parent support). In this case, ‘postExecute()’ is used to clean up the container cgroups created by YARN and ‘reapContainer()’ could be used to clean up/remove the container itself.
        I hope that helps.

        Show
        sidharta-s Sidharta Seethana added a comment - YARN-3853 added the ContainerRuntime interface. The objective of adding the ‘prepareContainer()’ and ‘reapContainer()’ methods was to provide finer grained control of the container lifecycle (and at some point add corresponding instrumentation to track where time is spent in the container lifecycle). Using docker containers as an example, ‘prepareContainer()’ could be hooked into docker ‘create’ (and maybe even image localization). reapContainer() could be used for (optional) post complete container deletion. Once container runtimes were introduced, the interaction with resource handlers got a bit … trickier. Right now, the same cgroups based resource handlers can be used across the ‘default’ and ‘docker’ container runtimes (mainly due to docker’s cgroup-parent support). In this case, ‘postExecute()’ is used to clean up the container cgroups created by YARN and ‘reapContainer()’ could be used to clean up/remove the container itself. I hope that helps.
        Hide
        vvasudev Varun Vasudev added a comment -

        1) Currently the ContainerRuntime.prepareContainer doesn't appear to have any usages in the standard execution of any containers. LinuxContainerExecutor.writeLaunchEnv is passed all of the information necessary to prepare the container runtime, and by overriding the method any modifications made to the run command will be persisted to the launch file.

        I did not realise that. cc Sidharta Seethana who wrote that code - it looks like we don't call prepareContainer anywhere; where did you originally mean for it to be used?

        Show
        vvasudev Varun Vasudev added a comment - 1) Currently the ContainerRuntime.prepareContainer doesn't appear to have any usages in the standard execution of any containers. LinuxContainerExecutor.writeLaunchEnv is passed all of the information necessary to prepare the container runtime, and by overriding the method any modifications made to the run command will be persisted to the launch file. I did not realise that. cc Sidharta Seethana who wrote that code - it looks like we don't call prepareContainer anywhere; where did you originally mean for it to be used?
        Hide
        gphillips Greg Phillips added a comment -

        Thanks for reviewing the patch Varun Vasudev.
        In effect this patch generates a temporary file (the java policy file) and modifies the container command to use that file on a container by container basis. To accomplish this using the ContainerRuntime interface, the prepareContainer method seemed to be the best option considering we want to modify the command before it is written to file.

        1) Currently the ContainerRuntime.prepareContainer doesn't appear to have any usages in the standard execution of any containers. LinuxContainerExecutor.writeLaunchEnv is passed all of the information necessary to prepare the container runtime, and by overriding the method any modifications made to the run command will be persisted to the launch file.

        2) For similar reasons to #1, the LinuxContainerExecutor seems to be the only class to use the LinuxContainerRuntime interface. The cleanup section of the ContainerExecutor.launchContainer includes:

            } finally {
              resourcesHandler.postExecute(containerId);
              try {
                linuxContainerRuntime.reapContainer(runtimeContext);
        

        The postExecute() method appears to share a similar utility to the ContainerRuntime.reapContainer.

        3) This patch proposes prepareContainer is executed in writeLaunchEnv. WriteLaunchEnv is not provided a context with the container which prevents us from using the ContainerRuntimeContext builder.

        To remedy these concerns we can do one of the following:
        1. Find a different location for prepareContainer which executes prior to the execution environment being written
        2. Accept the modification to the prepareContainer interface since it is still in alpha/unstable and is currently unused
        3. Create an additional LinuxContainerExecutor which adds ~4 lines to the original (though this doesn't resolve the issue of prepareContainer/reapContainer never being executed in 3.0.0-alpha1).

        Thanks again for reviewing this patch. I'm interested in your thoughts on the next steps for this effort. Additionally, I have another patch available which resolves the testing errors in the previous jenkins run, and removes the use of YARN queues (i.e. no changes to the MrAppMaster).

        Show
        gphillips Greg Phillips added a comment - Thanks for reviewing the patch Varun Vasudev . In effect this patch generates a temporary file (the java policy file) and modifies the container command to use that file on a container by container basis. To accomplish this using the ContainerRuntime interface, the prepareContainer method seemed to be the best option considering we want to modify the command before it is written to file. 1) Currently the ContainerRuntime.prepareContainer doesn't appear to have any usages in the standard execution of any containers. LinuxContainerExecutor.writeLaunchEnv is passed all of the information necessary to prepare the container runtime, and by overriding the method any modifications made to the run command will be persisted to the launch file. 2) For similar reasons to #1, the LinuxContainerExecutor seems to be the only class to use the LinuxContainerRuntime interface. The cleanup section of the ContainerExecutor.launchContainer includes: } finally { resourcesHandler.postExecute(containerId); try { linuxContainerRuntime.reapContainer(runtimeContext); The postExecute() method appears to share a similar utility to the ContainerRuntime.reapContainer. 3) This patch proposes prepareContainer is executed in writeLaunchEnv. WriteLaunchEnv is not provided a context with the container which prevents us from using the ContainerRuntimeContext builder. To remedy these concerns we can do one of the following: 1. Find a different location for prepareContainer which executes prior to the execution environment being written 2. Accept the modification to the prepareContainer interface since it is still in alpha/unstable and is currently unused 3. Create an additional LinuxContainerExecutor which adds ~4 lines to the original (though this doesn't resolve the issue of prepareContainer/reapContainer never being executed in 3.0.0-alpha1). Thanks again for reviewing this patch. I'm interested in your thoughts on the next steps for this effort. Additionally, I have another patch available which resolves the testing errors in the previous jenkins run, and removes the use of YARN queues (i.e. no changes to the MrAppMaster).
        Hide
        vvasudev Varun Vasudev added a comment -

        Thanks for the patch Greg Phillips. My apologies for the late comments -
        1)

           @Override
        +  public void writeLaunchEnv(OutputStream out, Map<String, String> environment,
        +      Map<Path, List<String>> resources, List<String> command, Path logDir,
        +      String user) throws IOException {
        +    try {
        +      linuxContainerRuntime.prepareContainer(environment, resources, command);
        +    } catch (ContainerExecutionException e) {
        +      throw new IOException("Unable to prepare container: ", e);
        +    }
        +    super.writeLaunchEnv(out, environment, resources, command, logDir, user);
        +  }
        +
        

        Can you please explain why you need this block? prepareContainer is really not meant to be called as part of the writeLaunchEnv

        2)

        +        linuxContainerRuntime.reapContainer(runtimeContext);
        

        Similar to the above - any reason why you’re calling reapContainer as part of the launchContainer call?

        3)

        -  public void prepareContainer(ContainerRuntimeContext ctx)
        +  public void prepareContainer(Map<String, String> environment,
        +      Map<Path, List<String>> resources, List<String> command)
               throws ContainerExecutionException {
             //nothing to do here at the moment.
           }
        

        Please don’t change these interfaces. ContainerExecutor interfaces are a public interface to allow users to plug their own implementations. If some field is missing, please add it to the context.

        Show
        vvasudev Varun Vasudev added a comment - Thanks for the patch Greg Phillips . My apologies for the late comments - 1) @Override + public void writeLaunchEnv(OutputStream out, Map< String , String > environment, + Map<Path, List< String >> resources, List< String > command, Path logDir, + String user) throws IOException { + try { + linuxContainerRuntime.prepareContainer(environment, resources, command); + } catch (ContainerExecutionException e) { + throw new IOException( "Unable to prepare container: " , e); + } + super .writeLaunchEnv(out, environment, resources, command, logDir, user); + } + Can you please explain why you need this block? prepareContainer is really not meant to be called as part of the writeLaunchEnv 2) + linuxContainerRuntime.reapContainer(runtimeContext); Similar to the above - any reason why you’re calling reapContainer as part of the launchContainer call? 3) - public void prepareContainer(ContainerRuntimeContext ctx) + public void prepareContainer(Map< String , String > environment, + Map<Path, List< String >> resources, List< String > command) throws ContainerExecutionException { //nothing to do here at the moment. } Please don’t change these interfaces. ContainerExecutor interfaces are a public interface to allow users to plug their own implementations. If some field is missing, please add it to the context.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 13s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 1m 38s Maven dependency ordering for branch
        +1 mvninstall 8m 53s trunk passed
        +1 compile 11m 41s trunk passed
        +1 checkstyle 1m 42s trunk passed
        +1 mvnsite 1m 52s trunk passed
        +1 mvneclipse 1m 10s trunk passed
        +1 findbugs 2m 55s trunk passed
        +1 javadoc 1m 27s trunk passed
        0 mvndep 0m 17s Maven dependency ordering for patch
        +1 mvninstall 1m 17s the patch passed
        +1 compile 10m 22s the patch passed
        -1 javac 10m 22s root generated 10 new + 694 unchanged - 0 fixed = 704 total (was 694)
        -0 checkstyle 1m 44s root: The patch generated 3 new + 442 unchanged - 3 fixed = 445 total (was 445)
        +1 mvnsite 2m 1s the patch passed
        +1 mvneclipse 1m 8s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 3m 27s the patch passed
        +1 javadoc 1m 33s the patch passed
        -1 unit 0m 41s hadoop-yarn-api in the patch failed.
        -1 unit 14m 58s hadoop-yarn-server-nodemanager in the patch failed.
        +1 unit 9m 12s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 53s The patch does not generate ASF License warnings.
        103m 7s



        Reason Tests
        Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields
          hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestJavaSandboxLinuxContainerRuntime
          hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:e809691
        JIRA Issue YARN-5280
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837822/YARN-5280.004.patch
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux 0436983a5cff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / de3b4aa
        Default Java 1.8.0_111
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/13811/artifact/patchprocess/diff-compile-javac-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13811/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13811/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13811/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13811/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13811/console
        Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 1m 38s Maven dependency ordering for branch +1 mvninstall 8m 53s trunk passed +1 compile 11m 41s trunk passed +1 checkstyle 1m 42s trunk passed +1 mvnsite 1m 52s trunk passed +1 mvneclipse 1m 10s trunk passed +1 findbugs 2m 55s trunk passed +1 javadoc 1m 27s trunk passed 0 mvndep 0m 17s Maven dependency ordering for patch +1 mvninstall 1m 17s the patch passed +1 compile 10m 22s the patch passed -1 javac 10m 22s root generated 10 new + 694 unchanged - 0 fixed = 704 total (was 694) -0 checkstyle 1m 44s root: The patch generated 3 new + 442 unchanged - 3 fixed = 445 total (was 445) +1 mvnsite 2m 1s the patch passed +1 mvneclipse 1m 8s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 3m 27s the patch passed +1 javadoc 1m 33s the patch passed -1 unit 0m 41s hadoop-yarn-api in the patch failed. -1 unit 14m 58s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 9m 12s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 53s The patch does not generate ASF License warnings. 103m 7s Reason Tests Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields   hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestJavaSandboxLinuxContainerRuntime   hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks Subsystem Report/Notes Docker Image:yetus/hadoop:e809691 JIRA Issue YARN-5280 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12837822/YARN-5280.004.patch Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 0436983a5cff 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / de3b4aa Default Java 1.8.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13811/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13811/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13811/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13811/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13811/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13811/console Powered by Apache Yetus 0.4.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
        Hide
        gphillips Greg Phillips added a comment -

        Modified sandbox-mode to again use three operating modes. Added tests for enforcing and permissive operating modes. With the sandbox mode enabled the JavaSandboxLinuxContainerRuntime will no longer sanitize container commands, instead will throw an exception in enforcing mode if a chained shell command (using '||' or '&&') is used. In permissive mode the container command will not be modified if it contains a chained shell command and no exception will be thrown.

        Show
        gphillips Greg Phillips added a comment - Modified sandbox-mode to again use three operating modes. Added tests for enforcing and permissive operating modes. With the sandbox mode enabled the JavaSandboxLinuxContainerRuntime will no longer sanitize container commands, instead will throw an exception in enforcing mode if a chained shell command (using '||' or '&&') is used. In permissive mode the container command will not be modified if it contains a chained shell command and no exception will be thrown.
        Hide
        gphillips Greg Phillips added a comment -
        • Removed sandbox modes, now just true/false for enabled/disabled.
          • Permissive mode unnecessary since ApplicationMaster containers run within security manager, so any ApplicationMaster needs to be trusted to function.
        • Refactored yarn configurations into YarnConfiguration
        • Updated documetation
        • Updated unit tests
        Show
        gphillips Greg Phillips added a comment - Removed sandbox modes, now just true/false for enabled/disabled. Permissive mode unnecessary since ApplicationMaster containers run within security manager, so any ApplicationMaster needs to be trusted to function. Refactored yarn configurations into YarnConfiguration Updated documetation Updated unit tests
        Hide
        gphillips Greg Phillips added a comment - - edited

        Robert Kanter - Thank you for reviewing the patch.

        1. The test failures didn't appear in my local unit testing, I was able to pull the logs from jenkins and I am attempting to track down the issue. The checkstyle and warnings mostly relate to using proprietary API's for the java policy framework. There are a handful of other examples of this warning in the total hadoop build, though I could find a way to work around using them if necessary.
        2. The queue name is used to whitelist an application so that it doesn't run with the security manager enabled. I've investigated several mechanisms for creating this whitelist behavior, and using queues offered access control and the correct scope. This does mean AM implementations will need to set this property in order for whitelisting to work (and currently only MR has this set). I am definitely interested in ideas for other ways of whitelisting applications.
        3. - 6. I will have an update including these changes in the next patch.
        Show
        gphillips Greg Phillips added a comment - - edited Robert Kanter - Thank you for reviewing the patch. The test failures didn't appear in my local unit testing, I was able to pull the logs from jenkins and I am attempting to track down the issue. The checkstyle and warnings mostly relate to using proprietary API's for the java policy framework. There are a handful of other examples of this warning in the total hadoop build, though I could find a way to work around using them if necessary. The queue name is used to whitelist an application so that it doesn't run with the security manager enabled. I've investigated several mechanisms for creating this whitelist behavior, and using queues offered access control and the correct scope. This does mean AM implementations will need to set this property in order for whitelisting to work (and currently only MR has this set). I am definitely interested in ideas for other ways of whitelisting applications. - 6. I will have an update including these changes in the next patch.
        Hide
        rkanter Robert Kanter added a comment -

        Thanks for continuing your work on this Greg Phillips. Here's some more feedback on the latest patch. I haven't had the time to test it out, so this is all based on reading through the code changes:

        1. Can you look into the test failures reported above? Also the checkstyle and warnings. Unfortunately, it looks like the Jenkins job has been purged so we don't have that info there anymore.
        2. Why do we add the queue name to the env? It looks like you're only using the queue in the JavaSandboxLinuxContainerRuntime, so I think it could go in the ContainerRuntimeContext instead.
          • Also, it's in MR code, so it's only going to be added for MR Apps and not other JVM-based Apps (e.g. Spark, Oozie-on-Yarn Launcher, etc).
        3. The class Javadoc comment in DelegatingLinuxContainerRuntime should be updated now that it can also delegate to the JavaSandboxLinuxContainerRuntime.
        4. The config properties added to JavaSandboxLinuxContainerRuntime (i.e. "yarn.nodemanager.linux-container-executor.sandbox-mode.*") should be defined in YarnConfiguration along with a default value. See the other properties in YarnConfiguration for examples.
        5. Instead of inlining PosixFilePermissions.fromString("rwxr-xr-x")) and similar in JavaSandboxLinuxContainerRuntime, they should be declared as private constants.
        6. We could use some additional unit tests. There's some complicated regexes, different operating modes, etc that we should make sure to more fully cover.
        Show
        rkanter Robert Kanter added a comment - Thanks for continuing your work on this Greg Phillips . Here's some more feedback on the latest patch. I haven't had the time to test it out, so this is all based on reading through the code changes: Can you look into the test failures reported above? Also the checkstyle and warnings. Unfortunately, it looks like the Jenkins job has been purged so we don't have that info there anymore. Why do we add the queue name to the env? It looks like you're only using the queue in the JavaSandboxLinuxContainerRuntime , so I think it could go in the ContainerRuntimeContext instead. Also, it's in MR code, so it's only going to be added for MR Apps and not other JVM-based Apps (e.g. Spark, Oozie-on-Yarn Launcher, etc). The class Javadoc comment in DelegatingLinuxContainerRuntime should be updated now that it can also delegate to the JavaSandboxLinuxContainerRuntime . The config properties added to JavaSandboxLinuxContainerRuntime (i.e. "yarn.nodemanager.linux-container-executor.sandbox-mode.*" ) should be defined in YarnConfiguration along with a default value. See the other properties in YarnConfiguration for examples. Instead of inlining PosixFilePermissions.fromString("rwxr-xr-x")) and similar in JavaSandboxLinuxContainerRuntime , they should be declared as private constants. We could use some additional unit tests. There's some complicated regexes, different operating modes, etc that we should make sure to more fully cover.
        Hide
        hadoopqa Hadoop QA added a comment -
        -1 overall



        Vote Subsystem Runtime Comment
        0 reexec 0m 26s Docker mode activated.
        +1 @author 0m 0s The patch does not contain any @author tags.
        +1 test4tests 0m 0s The patch appears to include 1 new or modified test files.
        0 mvndep 0m 16s Maven dependency ordering for branch
        +1 mvninstall 6m 58s trunk passed
        +1 compile 6m 53s trunk passed
        +1 checkstyle 1m 29s trunk passed
        +1 mvnsite 0m 55s trunk passed
        +1 mvneclipse 0m 29s trunk passed
        +1 findbugs 1m 15s trunk passed
        +1 javadoc 0m 33s trunk passed
        0 mvndep 0m 14s Maven dependency ordering for patch
        +1 mvninstall 0m 43s the patch passed
        +1 compile 6m 48s the patch passed
        -1 javac 6m 48s root generated 10 new + 708 unchanged - 0 fixed = 718 total (was 708)
        -1 checkstyle 1m 28s root: The patch generated 3 new + 240 unchanged - 2 fixed = 243 total (was 242)
        +1 mvnsite 0m 55s the patch passed
        +1 mvneclipse 0m 28s the patch passed
        +1 whitespace 0m 0s The patch has no whitespace issues.
        +1 findbugs 1m 32s the patch passed
        +1 javadoc 0m 32s the patch passed
        -1 unit 14m 29s hadoop-yarn-server-nodemanager in the patch failed.
        +1 unit 8m 32s hadoop-mapreduce-client-app in the patch passed.
        +1 asflicense 0m 23s The patch does not generate ASF License warnings.
        56m 7s



        Reason Tests
        Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager
          hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks



        Subsystem Report/Notes
        Docker Image:yetus/hadoop:9560f25
        JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830291/YARN-5280.002.patch
        JIRA Issue YARN-5280
        Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
        uname Linux d059ddaaff9c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
        Build tool maven
        Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
        git revision trunk / 14a696f
        Default Java 1.8.0_101
        findbugs v3.0.0
        javac https://builds.apache.org/job/PreCommit-YARN-Build/13214/artifact/patchprocess/diff-compile-javac-root.txt
        checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13214/artifact/patchprocess/diff-checkstyle-root.txt
        unit https://builds.apache.org/job/PreCommit-YARN-Build/13214/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13214/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
        Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13214/testReport/
        modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: .
        Console output https://builds.apache.org/job/PreCommit-YARN-Build/13214/console
        Powered by Apache Yetus 0.3.0 http://yetus.apache.org

        This message was automatically generated.

        Show
        hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 26s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 1 new or modified test files. 0 mvndep 0m 16s Maven dependency ordering for branch +1 mvninstall 6m 58s trunk passed +1 compile 6m 53s trunk passed +1 checkstyle 1m 29s trunk passed +1 mvnsite 0m 55s trunk passed +1 mvneclipse 0m 29s trunk passed +1 findbugs 1m 15s trunk passed +1 javadoc 0m 33s trunk passed 0 mvndep 0m 14s Maven dependency ordering for patch +1 mvninstall 0m 43s the patch passed +1 compile 6m 48s the patch passed -1 javac 6m 48s root generated 10 new + 708 unchanged - 0 fixed = 718 total (was 708) -1 checkstyle 1m 28s root: The patch generated 3 new + 240 unchanged - 2 fixed = 243 total (was 242) +1 mvnsite 0m 55s the patch passed +1 mvneclipse 0m 28s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 32s the patch passed +1 javadoc 0m 32s the patch passed -1 unit 14m 29s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 8m 32s hadoop-mapreduce-client-app in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 56m 7s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.containermanager.queuing.TestQueuingContainerManager   hadoop.yarn.server.nodemanager.TestLinuxContainerExecutorWithMocks Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12830291/YARN-5280.002.patch JIRA Issue YARN-5280 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d059ddaaff9c 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 14a696f Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13214/artifact/patchprocess/diff-compile-javac-root.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/13214/artifact/patchprocess/diff-checkstyle-root.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/13214/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/13214/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13214/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app U: . Console output https://builds.apache.org/job/PreCommit-YARN-Build/13214/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
        Hide
        gphillips Greg Phillips added a comment -

        This latest patch includes the controls discussed in this thread and uses the new ContainerRuntime API.
        Robert Kanter, Larry McCay, Vinod Kumar Vavilapalli - any thoughts on next steps?

        Show
        gphillips Greg Phillips added a comment - This latest patch includes the controls discussed in this thread and uses the new ContainerRuntime API. Robert Kanter , Larry McCay , Vinod Kumar Vavilapalli - any thoughts on next steps?
        Hide
        gphillips Greg Phillips added a comment -

        This patch is built against branch-3.0.0-alpha1 and adds a new LinuxContainerRuntime similar to the docker implementation. App-by-app whitelisting is now available via application queues.
        The following configurations are now available to modify the behavior of the JVMSandboxContainerRuntime:

        yarn.nodemanager.linux-container-executor.sandbox-mode : This yarn-site.xml setting has three options:

        • disabled - Default behavior. JavaSandboxLinuxContainerRuntime is disabled
        • permissive - JVM containers will run with Java Security Manager enabled. Non-JVM containers will run normally
        • enforcing - JVM containers will run with Java Security Manager enabled. Non-JVM containers will be prevented from executing and an ContainerExecutionException will be thrown

        yarn.nodemanager.linux-container-executor.sandbox-mode.file.permissions : Determines the file permissions for the application directories. The permissions come in the form of comma separated values (e.g. read,write,execute,delete). Defaults to read for read-only.

        yarn.nodemanager.linux-container-executor.sandbox-mode.policy : Accepts canonical path to a java policy file on the local filesystem. This file will be loaded as the base policy, any additional container grants will be appended to this base file. If not specified, the default java.policy file provided in the jar resources will be used.

        yarn.nodemanager.linux-container-executor.sandbox-mode.queue : Optional setting to specify a YARN queue which will be exempt from the sand-boxing process.

        Show
        gphillips Greg Phillips added a comment - This patch is built against branch-3.0.0-alpha1 and adds a new LinuxContainerRuntime similar to the docker implementation. App-by-app whitelisting is now available via application queues. The following configurations are now available to modify the behavior of the JVMSandboxContainerRuntime: yarn.nodemanager.linux-container-executor.sandbox-mode : This yarn-site.xml setting has three options: disabled - Default behavior. JavaSandboxLinuxContainerRuntime is disabled permissive - JVM containers will run with Java Security Manager enabled. Non-JVM containers will run normally enforcing - JVM containers will run with Java Security Manager enabled. Non-JVM containers will be prevented from executing and an ContainerExecutionException will be thrown yarn.nodemanager.linux-container-executor.sandbox-mode.file.permissions : Determines the file permissions for the application directories. The permissions come in the form of comma separated values (e.g. read,write,execute,delete). Defaults to read for read-only. yarn.nodemanager.linux-container-executor.sandbox-mode.policy : Accepts canonical path to a java policy file on the local filesystem. This file will be loaded as the base policy, any additional container grants will be appended to this base file. If not specified, the default java.policy file provided in the jar resources will be used. yarn.nodemanager.linux-container-executor.sandbox-mode.queue : Optional setting to specify a YARN queue which will be exempt from the sand-boxing process.
        Hide
        gphillips Greg Phillips added a comment -

        There are a few production environments which wish to provide Hadoop processing as a service. There have been instances where this model has lead to cluster instability, especially when the user base is very large. This optional feature is meant to provide controls to YARN to complement and harden the limitations imposed on containers.

        Show
        gphillips Greg Phillips added a comment - There are a few production environments which wish to provide Hadoop processing as a service. There have been instances where this model has lead to cluster instability, especially when the user base is very large. This optional feature is meant to provide controls to YARN to complement and harden the limitations imposed on containers.
        Hide
        lmccay Larry McCay added a comment -

        In order to prevent users from granting themselves excess permissions this would likely need to take the form of server side configurations.

        To clarify, the idea isn't so that applications would grant themselves permissions but instead declare the required permissions for the application. This allows for deployment time failure as apposed to runtime failure when a privileged action is attempted and fails. Of course, there is nothing stating that there couldn't be server side configuration to allow for a minimum set of permissions and some room for certain permissions that can be granted upon demand. In general, it would be expected that it would be a deploy time compare of those permissions required for deployment and those being granted by the container policy in server config.

        The jar signing subtasks certainly seem appropriate. I would still like to hear the driving usecase/s and how many folks actually need it.

        Show
        lmccay Larry McCay added a comment - In order to prevent users from granting themselves excess permissions this would likely need to take the form of server side configurations. To clarify, the idea isn't so that applications would grant themselves permissions but instead declare the required permissions for the application. This allows for deployment time failure as apposed to runtime failure when a privileged action is attempted and fails. Of course, there is nothing stating that there couldn't be server side configuration to allow for a minimum set of permissions and some room for certain permissions that can be granted upon demand. In general, it would be expected that it would be a deploy time compare of those permissions required for deployment and those being granted by the container policy in server config. The jar signing subtasks certainly seem appropriate. I would still like to hear the driving usecase/s and how many folks actually need it.
        Hide
        gphillips Greg Phillips added a comment -

        Hello Larry McCay - Thanks for the link to the EE specification for application permission requests. Given the range of frameworks that use YARN there is definitely utility in creating framework level rulesets. In order to prevent users from granting themselves excess permissions this would likely need to take the form of server side configurations. Thus far this effort has entailed providing all permissions to trusted code such as core hadoop libraries and surrounding projects (Pig, Hive, Oozie, etc.) while limiting privileges to the user contributed code that performs the processing. I would be interested to see if we could adopt a similar model for Slider; full privileges for the core libraries while locking down the user code. Initially I would like to prove this feature against MapReduce and the frameworks that leverage it. Additionally the solution must be extensible enough so other YARN frameworks can be handled differently by the NodeManager: either by disabling the security manager, or by providing a different set of permissions.

        In secure installations of Hadoop the creation and management of keystores is already a necessity. I have written some prototype utilities which streamline the process of signing Hadoop libraries. For Pig and Hive the dynamically created jars will need to be broken out. I have a test build of Pig which instead of creating an UberJar adds the necessary libs to tmpjars. This allows the libraries to maintain their signatures, and ultimately decreases the overhead of running Pig jobs since the broken out libraries will now be able to exist in the filecache. If this seems like an appropriate path I will create the subtasks for Hive and Pig.

        Show
        gphillips Greg Phillips added a comment - Hello Larry McCay - Thanks for the link to the EE specification for application permission requests. Given the range of frameworks that use YARN there is definitely utility in creating framework level rulesets. In order to prevent users from granting themselves excess permissions this would likely need to take the form of server side configurations. Thus far this effort has entailed providing all permissions to trusted code such as core hadoop libraries and surrounding projects (Pig, Hive, Oozie, etc.) while limiting privileges to the user contributed code that performs the processing. I would be interested to see if we could adopt a similar model for Slider; full privileges for the core libraries while locking down the user code. Initially I would like to prove this feature against MapReduce and the frameworks that leverage it. Additionally the solution must be extensible enough so other YARN frameworks can be handled differently by the NodeManager: either by disabling the security manager, or by providing a different set of permissions. In secure installations of Hadoop the creation and management of keystores is already a necessity. I have written some prototype utilities which streamline the process of signing Hadoop libraries. For Pig and Hive the dynamically created jars will need to be broken out. I have a test build of Pig which instead of creating an UberJar adds the necessary libs to tmpjars. This allows the libraries to maintain their signatures, and ultimately decreases the overhead of running Pig jobs since the broken out libraries will now be able to exist in the filecache. If this seems like an appropriate path I will create the subtasks for Hive and Pig.
        Hide
        lmccay Larry McCay added a comment -

        Hi Greg Phillips - having just read your pdf here, it has reminded me of some work that I was involved with for EE 7 - not sure where it went since I left that gig but I am curious about how an application might declare its need for particular permissions.

        https://java.net/downloads/javaee-spec/ee-sec-mgr-00-ljm.pdf see the section called EE 6.2.2.Y Declaring Permissions required by Application Components.

        In particular, I have Slider based application launches in mind where we do have an application descriptor where such hints/requests could be made at deployment time.

        As mentioned by Robert Kanter and your document, I do see challenges in the code signing bit.

        Have you seen significant push back from folks in the govt sector for requiring security manager?
        That has traditionally been the user base that really required it but I thought that I had sensed a bit of back off there.

        Show
        lmccay Larry McCay added a comment - Hi Greg Phillips - having just read your pdf here, it has reminded me of some work that I was involved with for EE 7 - not sure where it went since I left that gig but I am curious about how an application might declare its need for particular permissions. https://java.net/downloads/javaee-spec/ee-sec-mgr-00-ljm.pdf see the section called EE 6.2.2.Y Declaring Permissions required by Application Components. In particular, I have Slider based application launches in mind where we do have an application descriptor where such hints/requests could be made at deployment time. As mentioned by Robert Kanter and your document, I do see challenges in the code signing bit. Have you seen significant push back from folks in the govt sector for requiring security manager? That has traditionally been the user base that really required it but I thought that I had sensed a bit of back off there.
        Hide
        gphillips Greg Phillips added a comment -

        Larry McCay The current patch includes a default security policy which prevents all socket access, file access (except those explicitly whitelisted by the nodemanager), and private member access to untrusted code. It has been tested against Hadoop 2.6.4 successfully, and it will work with Kerberos enabled.

        Show
        gphillips Greg Phillips added a comment - Larry McCay The current patch includes a default security policy which prevents all socket access, file access (except those explicitly whitelisted by the nodemanager), and private member access to untrusted code. It has been tested against Hadoop 2.6.4 successfully, and it will work with Kerberos enabled.
        Hide
        lmccay Larry McCay added a comment -

        From my java platform perspective, I'm happy to see this thinking. From a Hadoop ops perspective, it sounds like a nightmare.
        It is possible that containers may make the headaches of security manager easier to deal with but there are generally many.

        I would suggest treading lightly at first and test access to files, sockets, etc and see where the pain rises.

        Show
        lmccay Larry McCay added a comment - From my java platform perspective, I'm happy to see this thinking. From a Hadoop ops perspective, it sounds like a nightmare. It is possible that containers may make the headaches of security manager easier to deal with but there are generally many. I would suggest treading lightly at first and test access to files, sockets, etc and see where the pain rises.
        Hide
        gphillips Greg Phillips added a comment -

        Vinod Kumar Vavilapalli - It certainly seems reasonable to refactor this feature into a JVM container runtime. It is important however that this feature remains opt-in since it requires additional considerations for cluster administration.

        I've tested kerberos integration & native code execution successfully with the current patch. Additionally to Robert Kanter's point I have modified Pig & Hive slightly to add all resources to tmpjars instead of building an uberjar, which has enabled the ability to sign the jars and subsequently execute successfully within a security manager. I am still cleaning these patches, and will create new sub-tickets when they are ready.

        I will follow up with testing results on your last suggestion. The one potential challenge we may run into is controlling file access using this method.

        Show
        gphillips Greg Phillips added a comment - Vinod Kumar Vavilapalli - It certainly seems reasonable to refactor this feature into a JVM container runtime. It is important however that this feature remains opt-in since it requires additional considerations for cluster administration. I've tested kerberos integration & native code execution successfully with the current patch. Additionally to Robert Kanter 's point I have modified Pig & Hive slightly to add all resources to tmpjars instead of building an uberjar, which has enabled the ability to sign the jars and subsequently execute successfully within a security manager. I am still cleaning these patches, and will create new sub-tickets when they are ready. I will follow up with testing results on your last suggestion. The one potential challenge we may run into is controlling file access using this method.
        Hide
        vinodkv Vinod Kumar Vavilapalli added a comment -

        Today, YARN (RMs / NMs) don't know whether the containers run JVMs or not - and we should keep it that way.

        We've been talking about Container universes / run-times (YARN-3853), the right way to do this is to think of a JVM run-time that can wrap this functionality only for JVM based containers.

        Irrespective of that, I think a reasonable way to make progress on this is to first experiment this functionality on the apps' side - say MapReduce and then promote it into YARN. Besides the performance impact, there are a bunch of scenarios that need to be looked at in the context of security-managers - native code, kerberos integration etc.

        Is it possible to run experiments with MapReduce alone first? We can actually do this without any code changes - using distributed-cache to distribute files and mapreduce.admin.map.child.java.opts / mapreduce.admin.reduce.child.java.opts.

        Show
        vinodkv Vinod Kumar Vavilapalli added a comment - Today, YARN (RMs / NMs) don't know whether the containers run JVMs or not - and we should keep it that way. We've been talking about Container universes / run-times ( YARN-3853 ), the right way to do this is to think of a JVM run-time that can wrap this functionality only for JVM based containers. Irrespective of that, I think a reasonable way to make progress on this is to first experiment this functionality on the apps' side - say MapReduce and then promote it into YARN. Besides the performance impact, there are a bunch of scenarios that need to be looked at in the context of security-managers - native code, kerberos integration etc. Is it possible to run experiments with MapReduce alone first? We can actually do this without any code changes - using distributed-cache to distribute files and mapreduce.admin.map.child.java.opts / mapreduce.admin.reduce.child.java.opts.
        Hide
        rkanter Robert Kanter added a comment -

        Thanks for posting this Greg Phillips. It seems like an interesting way to lock things down.

        A few points of feedback:

        • I think we should add a separate config for enabling/disabling the restriction that all containers run a JVM. This way, cluster admins have the option of locking down Apps like MR but still being flexible enough to allow other Apps/Containers that are not JVM-based.
        • As you call out in the document, jar signing will be tricky if we need to also sign any downstream projects like Pig or for dynamically generated jars.

        Mike Yoder, Larry McCay, any thoughts on this?

        Show
        rkanter Robert Kanter added a comment - Thanks for posting this Greg Phillips . It seems like an interesting way to lock things down. A few points of feedback: I think we should add a separate config for enabling/disabling the restriction that all containers run a JVM. This way, cluster admins have the option of locking down Apps like MR but still being flexible enough to allow other Apps/Containers that are not JVM-based. As you call out in the document, jar signing will be tricky if we need to also sign any downstream projects like Pig or for dynamically generated jars. Mike Yoder , Larry McCay , any thoughts on this?
        Hide
        gphillips Greg Phillips added a comment -

        Included prototype patch & proposal

        Show
        gphillips Greg Phillips added a comment - Included prototype patch & proposal

          People

          • Assignee:
            gphillips Greg Phillips
            Reporter:
            gphillips Greg Phillips
          • Votes:
            3 Vote for this issue
            Watchers:
            18 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development