Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3611 Support Docker Containers In LinuxContainerExecutor
  3. YARN-6623

Add support to turn off launching privileged containers in the container-executor

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Blocker
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      Hide
      A change in configuration for launching Docker containers under YARN. Docker container capabilities, mounts, networks and allowing privileged container have to specified in the container-executor.cfg. By default, all of the above are turned off. This change will break existing setups launching Docker containers under YARN. Please refer to the Docker containers under YARN documentation for more information.
      Show
      A change in configuration for launching Docker containers under YARN. Docker container capabilities, mounts, networks and allowing privileged container have to specified in the container-executor.cfg. By default, all of the above are turned off. This change will break existing setups launching Docker containers under YARN. Please refer to the Docker containers under YARN documentation for more information.

      Description

      Currently, launching privileged containers is controlled by the NM. We should add a flag to the container-executor.cfg allowing admins to disable launching privileged containers at the container-executor level.

      1. YARN-6623.001.patch
        191 kB
        Varun Vasudev
      2. YARN-6623.002.patch
        197 kB
        Varun Vasudev
      3. YARN-6623.003.patch
        197 kB
        Varun Vasudev
      4. YARN-6623.004.patch
        204 kB
        Varun Vasudev
      5. YARN-6623.005.patch
        202 kB
        Varun Vasudev
      6. YARN-6623.006.patch
        202 kB
        Varun Vasudev
      7. YARN-6623.007.patch
        206 kB
        Varun Vasudev
      8. YARN-6623.008.patch
        206 kB
        Varun Vasudev
      9. YARN-6623.009.patch
        205 kB
        Varun Vasudev
      10. YARN-6623.010.patch
        208 kB
        Varun Vasudev
      11. YARN-6623.011.patch
        219 kB
        Varun Vasudev
      12. YARN-6623.012.patch
        217 kB
        Varun Vasudev
      13. YARN-6623.013.patch
        217 kB
        Varun Vasudev
      14. YARN-6623-branch-2.013.patch
        223 kB
        Varun Vasudev
      15. YARN-6623-branch-2.014.patch
        224 kB
        Varun Vasudev
      16. YARN-6623-branch-2.015.patch
        224 kB
        Varun Vasudev
      17. cetest.stderr
        3 kB
        Eric Yang
      18. cetest.stdout
        8 kB
        Eric Yang

        Issue Links

          Activity

          Hide
          andrew.wang Andrew Wang added a comment -

          Hi Varun, do you mind adding a release note for this JIRA, given that it's marked as incompatible? Thanks!

          Show
          andrew.wang Andrew Wang added a comment - Hi Varun, do you mind adding a release note for this JIRA, given that it's marked as incompatible? Thanks!
          Hide
          leftnoteasy Wangda Tan added a comment -

          Pushed to branch-2 as well, thanks Varun Vasudev and additional suggestions from Eric Yang/Eric Badger/Jian He!

          Show
          leftnoteasy Wangda Tan added a comment - Pushed to branch-2 as well, thanks Varun Vasudev and additional suggestions from Eric Yang / Eric Badger / Jian He !
          Hide
          leftnoteasy Wangda Tan added a comment -

          Attached patch LGTM, thanks Varun Vasudev.

          Show
          leftnoteasy Wangda Tan added a comment - Attached patch LGTM, thanks Varun Vasudev .
          Hide
          eyang Eric Yang added a comment -

          Varun Vasudev I opened YARN-7344 to track failed unit test issues on CentOS7.

          Show
          eyang Eric Yang added a comment - Varun Vasudev I opened YARN-7344 to track failed unit test issues on CentOS7.
          Hide
          vvasudev Varun Vasudev added a comment -

          Wangda Tan - can you take a look at the branch-2 patch please? Thanks!

          Show
          vvasudev Varun Vasudev added a comment - Wangda Tan - can you take a look at the branch-2 patch please? Thanks!
          Hide
          vvasudev Varun Vasudev added a comment -

          Eric Yang - the tests did run as part of the trunk pre-commit build. The test reports are lost now but it will appear under the section (root) in the test report.

          The tests seem to pass for me on the Mac and an Ubuntu VM. What's the environment you're using? Maybe you should open a new ticket with details?

          Here's the result I got -
          Mac

          Varuns-MacBook-Pro:hadoop-yarn-server-nodemanager varun$ mvn test -Dtest=cetest -Pnative
          [INFO] Scanning for projects...
          [INFO]
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Apache Hadoop YARN NodeManager 3.1.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO]
          [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-yarn-server-nodemanager ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO]
          [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) @ hadoop-yarn-server-nodemanager ---
          [INFO] No changes detected in protoc files, skipping generation.
          [INFO]
          [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-yarn-server-nodemanager ---
          [INFO]
          [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-yarn-server-nodemanager ---
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 4 resources
          [INFO] Copying 2 resources
          [INFO]
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-yarn-server-nodemanager ---
          [INFO] Compiling 3 source files to /Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
          [INFO]
          [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-compile (cmake-compile) @ hadoop-yarn-server-nodemanager ---
          [INFO] Running cmake /Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
          [INFO] with extra environment variables {
            CFLAGS = ''
          }
          [INFO] Running make -j 8 VERBOSE=1
          [INFO] cmake compilation finished successfully in 205 millisecond(s).
          [INFO]
          [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-yarn-server-nodemanager ---
          [INFO] Executing tasks
          
          main:
               [copy] Copying 4 files to /Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test
          [INFO] Executed tasks
          [INFO]
          [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hadoop-yarn-server-nodemanager ---
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 7 resources
          [INFO] Copying 2 resources
          [INFO]
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hadoop-yarn-server-nodemanager ---
          [INFO] Nothing to compile - all classes are up to date
          [INFO]
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-yarn-server-nodemanager ---
          [INFO]
          [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (test-container-executor) @ hadoop-yarn-server-nodemanager ---
          [INFO]
          [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (cetest) @ hadoop-yarn-server-nodemanager ---
          [INFO] -------------------------------------------------------
          [INFO]  C M A K E B U I L D E R    T E S T
          [INFO] -------------------------------------------------------
          [INFO] cetest: running /Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test/cetest --gtest_filter=-Perf. --gtest_output=xml:/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports/TEST-cetest.xml
          [INFO] with extra environment variables {}
          [INFO] STATUS: SUCCESS after 87 millisecond(s).
          [INFO] -------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 4.263 s
          [INFO] Finished at: 2017-10-17T15:43:32+05:30
          [INFO] Final Memory: 35M/318M
          [INFO] ------------------------------------------------------------------------
          

          Ubuntu 16

          varun@nm-u16:~/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager$ mvn test -Dtest=cetest -Pnative
          [INFO] Scanning for projects...
          [INFO]
          [INFO] ------------------------------------------------------------------------
          [INFO] Building Apache Hadoop YARN NodeManager 3.1.0-SNAPSHOT
          [INFO] ------------------------------------------------------------------------
          [INFO]
          [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-yarn-server-nodemanager ---
          [INFO] Executing tasks
          
          main:
          [INFO] Executed tasks
          [INFO]
          [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) @ hadoop-yarn-server-nodemanager ---
          [INFO] No changes detected in protoc files, skipping generation.
          [INFO]
          [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-yarn-server-nodemanager ---
          [INFO]
          [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-yarn-server-nodemanager ---
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 4 resources
          [INFO] Copying 2 resources
          [INFO]
          [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-yarn-server-nodemanager ---
          [INFO] Compiling 3 source files to /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes
          [INFO]
          [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-compile (cmake-compile) @ hadoop-yarn-server-nodemanager ---
          [INFO] Running cmake /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles
          [INFO] with extra environment variables {
            CFLAGS = ''
          }
          [INFO] Running make -j 1 VERBOSE=1
          [INFO] cmake compilation finished successfully in 365 millisecond(s).
          [INFO]
          [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-yarn-server-nodemanager ---
          [INFO] Executing tasks
          
          main:
               [copy] Copying 4 files to /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test
          [INFO] Executed tasks
          [INFO]
          [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hadoop-yarn-server-nodemanager ---
          [INFO] Using 'UTF-8' encoding to copy filtered resources.
          [INFO] Copying 7 resources
          [INFO] Copying 2 resources
          [INFO]
          [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hadoop-yarn-server-nodemanager ---
          [INFO] Compiling 29 source files to /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-classes
          [WARNING] /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java: /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java uses or overrides a deprecated API.
          [WARNING] /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java: Recompile with -Xlint:deprecation for details.
          [WARNING] /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestJavaSandboxLinuxContainerRuntime.java: Some input files use unchecked or unsafe operations.
          [WARNING] /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestJavaSandboxLinuxContainerRuntime.java: Recompile with -Xlint:unchecked for details.
          [INFO]
          [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-yarn-server-nodemanager ---
          [INFO]
          [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (test-container-executor) @ hadoop-yarn-server-nodemanager ---
          [INFO]
          [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (cetest) @ hadoop-yarn-server-nodemanager ---
          [INFO] -------------------------------------------------------
          [INFO]  C M A K E B U I L D E R    T E S T
          [INFO] -------------------------------------------------------
          [INFO] cetest: running /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test/cetest --gtest_filter=-Perf. --gtest_output=xml:/home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports/TEST-cetest.xml
          [INFO] with extra environment variables {}
          [INFO] STATUS: SUCCESS after 64 millisecond(s).
          [INFO] -------------------------------------------------------
          [INFO] ------------------------------------------------------------------------
          [INFO] BUILD SUCCESS
          [INFO] ------------------------------------------------------------------------
          [INFO] Total time: 11.906 s
          [INFO] Finished at: 2017-10-17T15:50:48+05:30
          [INFO] Final Memory: 38M/121M
          [INFO] ------------------------------------------------------------------------
          
          Show
          vvasudev Varun Vasudev added a comment - Eric Yang - the tests did run as part of the trunk pre-commit build. The test reports are lost now but it will appear under the section (root) in the test report. The tests seem to pass for me on the Mac and an Ubuntu VM. What's the environment you're using? Maybe you should open a new ticket with details? Here's the result I got - Mac Varuns-MacBook-Pro:hadoop-yarn-server-nodemanager varun$ mvn test -Dtest=cetest -Pnative [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Apache Hadoop YARN NodeManager 3.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-yarn-server-nodemanager --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) @ hadoop-yarn-server-nodemanager --- [INFO] No changes detected in protoc files, skipping generation. [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-yarn-server-nodemanager --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-yarn-server-nodemanager --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-yarn-server-nodemanager --- [INFO] Compiling 3 source files to /Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes [INFO] [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-compile (cmake-compile) @ hadoop-yarn-server-nodemanager --- [INFO] Running cmake /Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles [INFO] with extra environment variables { CFLAGS = '' } [INFO] Running make -j 8 VERBOSE=1 [INFO] cmake compilation finished successfully in 205 millisecond(s). [INFO] [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-yarn-server-nodemanager --- [INFO] Executing tasks main: [copy] Copying 4 files to /Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test [INFO] Executed tasks [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hadoop-yarn-server-nodemanager --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 7 resources [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hadoop-yarn-server-nodemanager --- [INFO] Nothing to compile - all classes are up to date [INFO] [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-yarn-server-nodemanager --- [INFO] [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (test-container-executor) @ hadoop-yarn-server-nodemanager --- [INFO] [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (cetest) @ hadoop-yarn-server-nodemanager --- [INFO] ------------------------------------------------------- [INFO] C M A K E B U I L D E R T E S T [INFO] ------------------------------------------------------- [INFO] cetest: running /Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test/cetest --gtest_filter=-Perf. --gtest_output=xml:/Users/varun/Workspace/apache/committer-rw/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports/TEST-cetest.xml [INFO] with extra environment variables {} [INFO] STATUS: SUCCESS after 87 millisecond(s). [INFO] ------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 4.263 s [INFO] Finished at: 2017-10-17T15:43:32+05:30 [INFO] Final Memory: 35M/318M [INFO] ------------------------------------------------------------------------ Ubuntu 16 varun@nm-u16:~/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager$ mvn test -Dtest=cetest -Pnative [INFO] Scanning for projects... [INFO] [INFO] ------------------------------------------------------------------------ [INFO] Building Apache Hadoop YARN NodeManager 3.1.0-SNAPSHOT [INFO] ------------------------------------------------------------------------ [INFO] [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-yarn-server-nodemanager --- [INFO] Executing tasks main: [INFO] Executed tasks [INFO] [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:protoc (compile-protoc) @ hadoop-yarn-server-nodemanager --- [INFO] No changes detected in protoc files, skipping generation. [INFO] [INFO] --- maven-remote-resources-plugin:1.5:process (default) @ hadoop-yarn-server-nodemanager --- [INFO] [INFO] --- maven-resources-plugin:2.6:resources (default-resources) @ hadoop-yarn-server-nodemanager --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 4 resources [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:compile (default-compile) @ hadoop-yarn-server-nodemanager --- [INFO] Compiling 3 source files to /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/classes [INFO] [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-compile (cmake-compile) @ hadoop-yarn-server-nodemanager --- [INFO] Running cmake /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -DHADOOP_CONF_DIR=../etc/hadoop -DJVM_ARCH_DATA_MODEL=64 -G Unix Makefiles [INFO] with extra environment variables { CFLAGS = '' } [INFO] Running make -j 1 VERBOSE=1 [INFO] cmake compilation finished successfully in 365 millisecond(s). [INFO] [INFO] --- maven-antrun-plugin:1.7:run (make) @ hadoop-yarn-server-nodemanager --- [INFO] Executing tasks main: [copy] Copying 4 files to /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test [INFO] Executed tasks [INFO] [INFO] --- maven-resources-plugin:2.6:testResources (default-testResources) @ hadoop-yarn-server-nodemanager --- [INFO] Using 'UTF-8' encoding to copy filtered resources. [INFO] Copying 7 resources [INFO] Copying 2 resources [INFO] [INFO] --- maven-compiler-plugin:3.1:testCompile (default-testCompile) @ hadoop-yarn-server-nodemanager --- [INFO] Compiling 29 source files to /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/test-classes [WARNING] /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java: /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java uses or overrides a deprecated API. [WARNING] /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/logaggregation/TestLogAggregationService.java: Recompile with -Xlint:deprecation for details. [WARNING] /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestJavaSandboxLinuxContainerRuntime.java: Some input files use unchecked or unsafe operations. [WARNING] /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestJavaSandboxLinuxContainerRuntime.java: Recompile with -Xlint:unchecked for details. [INFO] [INFO] --- maven-surefire-plugin:2.17:test (default-test) @ hadoop-yarn-server-nodemanager --- [INFO] [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (test-container-executor) @ hadoop-yarn-server-nodemanager --- [INFO] [INFO] --- hadoop-maven-plugins:3.1.0-SNAPSHOT:cmake-test (cetest) @ hadoop-yarn-server-nodemanager --- [INFO] ------------------------------------------------------- [INFO] C M A K E B U I L D E R T E S T [INFO] ------------------------------------------------------- [INFO] cetest: running /home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native/test/cetest --gtest_filter=-Perf. --gtest_output=xml:/home/varun/Workspace/apache/dev/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/surefire-reports/TEST-cetest.xml [INFO] with extra environment variables {} [INFO] STATUS: SUCCESS after 64 millisecond(s). [INFO] ------------------------------------------------------- [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 11.906 s [INFO] Finished at: 2017-10-17T15:50:48+05:30 [INFO] Final Memory: 38M/121M [INFO] ------------------------------------------------------------------------
          Hide
          eyang Eric Yang added a comment -

          Varun Vasudev Here are the output of the failed test. Any idea why the tests fail?

          Show
          eyang Eric Yang added a comment - Varun Vasudev Here are the output of the failed test. Any idea why the tests fail?
          Hide
          eyang Eric Yang added a comment -

          The current trunk seems to fail on these test cases:

          [  FAILED  ] 3 tests, listed below:
          [  FAILED  ] TestDockerUtil.test_check_mount_permitted
          [  FAILED  ] TestDockerUtil.test_normalize_mounts
          [  FAILED  ] TestDockerUtil.test_add_rw_mounts
          
          Show
          eyang Eric Yang added a comment - The current trunk seems to fail on these test cases: [ FAILED ] 3 tests, listed below: [ FAILED ] TestDockerUtil.test_check_mount_permitted [ FAILED ] TestDockerUtil.test_normalize_mounts [ FAILED ] TestDockerUtil.test_add_rw_mounts
          Hide
          eyang Eric Yang added a comment -

          Talked to Tan, Wangda and he said the test can be triggered using:

          -Dtest=cetest,test-container-executor -Pnative
          

          To the maven command. It would be nice if it is covered by default test without passing in test case name.

          Show
          eyang Eric Yang added a comment - Talked to Tan, Wangda and he said the test can be triggered using: -Dtest=cetest,test-container-executor -Pnative To the maven command. It would be nice if it is covered by default test without passing in test case name.
          Hide
          eyang Eric Yang added a comment -

          Varun Vasudev How is the C++ test case getting triggered? mvn clean test -Pnative doesn't trigger test_docker_util.cc to run.

          Show
          eyang Eric Yang added a comment - Varun Vasudev How is the C++ test case getting triggered? mvn clean test -Pnative doesn't trigger test_docker_util.cc to run.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 22s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                branch-2 Compile Tests
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 7m 49s branch-2 passed
          +1 compile 2m 47s branch-2 passed
          +1 checkstyle 0m 44s branch-2 passed
          +1 mvnsite 3m 53s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 59s branch-2 passed
          +1 javadoc 2m 22s branch-2 passed
                Patch Compile Tests
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 3m 9s the patch passed
          +1 compile 2m 49s the patch passed
          +1 cc 2m 49s hadoop-yarn-project_hadoop-yarn generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          +1 javac 2m 49s the patch passed
          +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 24 unchanged - 5 fixed = 24 total (was 29)
          +1 mvnsite 3m 56s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 1m 10s the patch passed
          +1 javadoc 2m 18s the patch passed
                Other Tests
          -1 unit 104m 13s hadoop-yarn in the patch failed.
          -1 unit 13m 58s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 0m 7s hadoop-yarn-site in the patch passed.
          -1 asflicense 0m 21s The patch generated 1 ASF License warnings.
          176m 15s



          Reason Tests
          Failed junit tests hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler
            hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler
          Timed out junit tests org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater
            org.apache.hadoop.yarn.server.resourcemanager.TestRMHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:eaf5c66
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892411/YARN-6623-branch-2.015.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall shadedclient findbugs checkstyle
          uname Linux 4265fe8cd48c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / d82edb4
          Default Java 1.7.0_151
          findbugs v3.0.0
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17957/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17957/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17957/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/17957/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17957/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 7m 49s branch-2 passed +1 compile 2m 47s branch-2 passed +1 checkstyle 0m 44s branch-2 passed +1 mvnsite 3m 53s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 59s branch-2 passed +1 javadoc 2m 22s branch-2 passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 3m 9s the patch passed +1 compile 2m 49s the patch passed +1 cc 2m 49s hadoop-yarn-project_hadoop-yarn generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) +1 javac 2m 49s the patch passed +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 24 unchanged - 5 fixed = 24 total (was 29) +1 mvnsite 3m 56s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 10s the patch passed +1 javadoc 2m 18s the patch passed       Other Tests -1 unit 104m 13s hadoop-yarn in the patch failed. -1 unit 13m 58s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 0m 7s hadoop-yarn-site in the patch passed. -1 asflicense 0m 21s The patch generated 1 ASF License warnings. 176m 15s Reason Tests Failed junit tests hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler   hadoop.yarn.server.nodemanager.scheduler.TestDistributedScheduler Timed out junit tests org.apache.hadoop.yarn.server.nodemanager.TestNodeStatusUpdater   org.apache.hadoop.yarn.server.resourcemanager.TestRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:eaf5c66 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892411/YARN-6623-branch-2.015.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall shadedclient findbugs checkstyle uname Linux 4265fe8cd48c 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / d82edb4 Default Java 1.7.0_151 findbugs v3.0.0 unit https://builds.apache.org/job/PreCommit-YARN-Build/17957/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17957/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17957/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/17957/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17957/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          New version(branch-2.015.patch) with compile fix.

          Show
          vvasudev Varun Vasudev added a comment - New version(branch-2.015.patch) with compile fix.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 11m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                branch-2 Compile Tests
          0 mvndep 0m 22s Maven dependency ordering for branch
          +1 mvninstall 7m 2s branch-2 passed
          +1 compile 2m 40s branch-2 passed
          +1 checkstyle 0m 44s branch-2 passed
          +1 mvnsite 3m 47s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 55s branch-2 passed
          +1 javadoc 2m 16s branch-2 passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          -1 mvninstall 1m 48s hadoop-yarn in the patch failed.
          -1 mvninstall 0m 25s hadoop-yarn-server-nodemanager in the patch failed.
          -1 compile 1m 29s hadoop-yarn in the patch failed.
          -1 cc 1m 29s hadoop-yarn in the patch failed.
          -1 javac 1m 29s hadoop-yarn in the patch failed.
          +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 24 unchanged - 5 fixed = 24 total (was 29)
          -1 mvnsite 1m 22s hadoop-yarn in the patch failed.
          -1 mvnsite 0m 26s hadoop-yarn-server-nodemanager in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          -1 findbugs 0m 17s hadoop-yarn-server-nodemanager in the patch failed.
          +1 javadoc 2m 14s the patch passed
                Other Tests
          -1 unit 55m 46s hadoop-yarn in the patch failed.
          -1 unit 0m 44s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 0m 9s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 22s The patch does not generate ASF License warnings.
          102m 21s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:eaf5c66
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892384/YARN-6623-branch-2.014.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall shadedclient findbugs checkstyle
          uname Linux 093a2c61ac5d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 4b734fe
          Default Java 1.7.0_151
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn.txt
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          cc https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17955/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17955/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 11m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 22s Maven dependency ordering for branch +1 mvninstall 7m 2s branch-2 passed +1 compile 2m 40s branch-2 passed +1 checkstyle 0m 44s branch-2 passed +1 mvnsite 3m 47s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 55s branch-2 passed +1 javadoc 2m 16s branch-2 passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch -1 mvninstall 1m 48s hadoop-yarn in the patch failed. -1 mvninstall 0m 25s hadoop-yarn-server-nodemanager in the patch failed. -1 compile 1m 29s hadoop-yarn in the patch failed. -1 cc 1m 29s hadoop-yarn in the patch failed. -1 javac 1m 29s hadoop-yarn in the patch failed. +1 checkstyle 0m 40s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 24 unchanged - 5 fixed = 24 total (was 29) -1 mvnsite 1m 22s hadoop-yarn in the patch failed. -1 mvnsite 0m 26s hadoop-yarn-server-nodemanager in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 17s hadoop-yarn-server-nodemanager in the patch failed. +1 javadoc 2m 14s the patch passed       Other Tests -1 unit 55m 46s hadoop-yarn in the patch failed. -1 unit 0m 44s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 0m 9s hadoop-yarn-site in the patch passed. +1 asflicense 0m 22s The patch does not generate ASF License warnings. 102m 21s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.TestAbstractYarnScheduler Subsystem Report/Notes Docker Image:yetus/hadoop:eaf5c66 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892384/YARN-6623-branch-2.014.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall shadedclient findbugs checkstyle uname Linux 093a2c61ac5d 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 4b734fe Default Java 1.7.0_151 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17955/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17955/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17955/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new version of the branch-2 patch to fix the compile error.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new version of the branch-2 patch to fix the compile error.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 21s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                branch-2 Compile Tests
          0 mvndep 0m 28s Maven dependency ordering for branch
          +1 mvninstall 6m 54s branch-2 passed
          +1 compile 2m 36s branch-2 passed
          +1 checkstyle 0m 39s branch-2 passed
          +1 mvnsite 3m 35s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 52s branch-2 passed
          +1 javadoc 2m 8s branch-2 passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          -1 mvninstall 1m 44s hadoop-yarn in the patch failed.
          -1 mvninstall 0m 20s hadoop-yarn-server-nodemanager in the patch failed.
          -1 compile 1m 13s hadoop-yarn in the patch failed.
          -1 cc 1m 13s hadoop-yarn in the patch failed.
          -1 javac 1m 13s hadoop-yarn in the patch failed.
          +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 24 unchanged - 5 fixed = 24 total (was 29)
          -1 mvnsite 1m 14s hadoop-yarn in the patch failed.
          -1 mvnsite 0m 18s hadoop-yarn-server-nodemanager in the patch failed.
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          -1 findbugs 0m 15s hadoop-yarn-server-nodemanager in the patch failed.
          +1 javadoc 2m 13s the patch passed
                Other Tests
          -1 unit 17m 45s hadoop-yarn in the patch failed.
          -1 unit 0m 18s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 0m 7s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          51m 27s



          Reason Tests
          Timed out junit tests org.apache.hadoop.yarn.webapp.TestWebApp



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:eaf5c66
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892353/YARN-6623-branch-2.013.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall shadedclient findbugs checkstyle
          uname Linux 47d0658a4c71 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 4b734fe
          Default Java 1.7.0_151
          findbugs v3.0.0
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn.txt
          mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          compile https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          cc https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn.txt
          mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17951/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17951/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 21s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       branch-2 Compile Tests 0 mvndep 0m 28s Maven dependency ordering for branch +1 mvninstall 6m 54s branch-2 passed +1 compile 2m 36s branch-2 passed +1 checkstyle 0m 39s branch-2 passed +1 mvnsite 3m 35s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 52s branch-2 passed +1 javadoc 2m 8s branch-2 passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch -1 mvninstall 1m 44s hadoop-yarn in the patch failed. -1 mvninstall 0m 20s hadoop-yarn-server-nodemanager in the patch failed. -1 compile 1m 13s hadoop-yarn in the patch failed. -1 cc 1m 13s hadoop-yarn in the patch failed. -1 javac 1m 13s hadoop-yarn in the patch failed. +1 checkstyle 0m 39s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 24 unchanged - 5 fixed = 24 total (was 29) -1 mvnsite 1m 14s hadoop-yarn in the patch failed. -1 mvnsite 0m 18s hadoop-yarn-server-nodemanager in the patch failed. +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site -1 findbugs 0m 15s hadoop-yarn-server-nodemanager in the patch failed. +1 javadoc 2m 13s the patch passed       Other Tests -1 unit 17m 45s hadoop-yarn in the patch failed. -1 unit 0m 18s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 0m 7s hadoop-yarn-site in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 51m 27s Reason Tests Timed out junit tests org.apache.hadoop.yarn.webapp.TestWebApp Subsystem Report/Notes Docker Image:yetus/hadoop:eaf5c66 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12892353/YARN-6623-branch-2.013.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall shadedclient findbugs checkstyle uname Linux 47d0658a4c71 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 4b734fe Default Java 1.7.0_151 findbugs v3.0.0 mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn.txt mvninstall https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-mvninstall-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt compile https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn.txt mvnsite https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-mvnsite-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17951/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17951/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17951/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Attached a patch for branch-2.

          Show
          vvasudev Varun Vasudev added a comment - Attached a patch for branch-2.
          Hide
          eyang Eric Yang added a comment -

          Eric Badger In some secure environment, I have seen that container-executor.cfg is set to non-world readable because the administrator doesn't want people to know about the allowed and bannded users on the cluster. Another possibility is the default umask are set to 027, when admin generates container-executor.cfg via Ambari. it is often set to world non-readable for secure environment.

          Show
          eyang Eric Yang added a comment - Eric Badger In some secure environment, I have seen that container-executor.cfg is set to non-world readable because the administrator doesn't want people to know about the allowed and bannded users on the cluster. Another possibility is the default umask are set to 027, when admin generates container-executor.cfg via Ambari. it is often set to world non-readable for secure environment.
          Hide
          ebadger Eric Badger added a comment -

          It can be deprecated or used for fast-failing containers. I'm open to either option.

          Is there a reason we can't make container-executor.cfg world readable so that we can parse it in the NM before the container-executor is launched and fail fast that way?

          Show
          ebadger Eric Badger added a comment - It can be deprecated or used for fast-failing containers. I'm open to either option. Is there a reason we can't make container-executor.cfg world readable so that we can parse it in the NM before the container-executor is launched and fail fast that way?
          Hide
          vvasudev Varun Vasudev added a comment -

          There's already a config in yarn-site.xml
          "yarn.nodemanager.runtime.linux.docker.allowed-container-networks"

          Will the one in yarn-site.xml be deprecated ? what's the story here?

          It can be deprecated or used for fast-failing containers. I'm open to either option.

          Also, should the default value include host and bridge like the yarn-defalut.xml did ? otherwise, the default setting will just not work, user has to explicitly chage it

          No, the idea is to have admins explicitly change it.

          Show
          vvasudev Varun Vasudev added a comment - There's already a config in yarn-site.xml "yarn.nodemanager.runtime.linux.docker.allowed-container-networks" Will the one in yarn-site.xml be deprecated ? what's the story here? It can be deprecated or used for fast-failing containers. I'm open to either option. Also, should the default value include host and bridge like the yarn-defalut.xml did ? otherwise, the default setting will just not work, user has to explicitly chage it No, the idea is to have admins explicitly change it.
          Hide
          jianhe Jian He added a comment -

          Got a question, for this config added
          docker.allowed.networks=## comma seperated networks that can be used. e.g bridge,host,none

          There's already a config in yarn-site.xml
          "yarn.nodemanager.runtime.linux.docker.allowed-container-networks"

          Will the one in yarn-site.xml be deprecated ? what's the story here?

          Also, should the default value include host and bridge like the yarn-defalut.xml did ? otherwise, the default setting will just not work, user has to explicitly chage it

          Show
          jianhe Jian He added a comment - Got a question, for this config added docker.allowed.networks=## comma seperated networks that can be used. e.g bridge,host,none There's already a config in yarn-site.xml "yarn.nodemanager.runtime.linux.docker.allowed-container-networks" Will the one in yarn-site.xml be deprecated ? what's the story here? Also, should the default value include host and bridge like the yarn-defalut.xml did ? otherwise, the default setting will just not work, user has to explicitly chage it
          Hide
          asuresh Arun Suresh added a comment -

          Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ?

          Show
          asuresh Arun Suresh added a comment - Is this still on target for 2.9.0 ? If not, can we we push this out to the next major release ?
          Hide
          andrew.wang Andrew Wang added a comment -

          Sorry, this missed the train for RC0. We'll pick it up for GA, or possibly if there's an RC1.

          Show
          andrew.wang Andrew Wang added a comment - Sorry, this missed the train for RC0. We'll pick it up for GA, or possibly if there's an RC1.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the reviews Eric Badger, Miklos Szegedi, Shane Kumpf, Sunil Govind and Wangda Tan!

          I'll get a patch together for branch-2.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the reviews Eric Badger , Miklos Szegedi , Shane Kumpf , Sunil Govind and Wangda Tan ! I'll get a patch together for branch-2.
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12994 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12994/)
          YARN-6623. Add support to turn off launching privileged containers in (wangda: rev d3b1c6319546706c41a2011ead6c3fe208883200)

          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerStopCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRmCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRunCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRunCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_util.cc
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommandExecutor.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.h
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerLoadCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerInspectCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.h
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/resources/test/test-configurations/docker-container-executor.cfg
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRmCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerLoadCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerStopCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/conf/container-executor.cfg
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.c
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/common/module-configs.c
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerClient.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/common/module-configs.h
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerPullCommand.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerInspectCommand.java
          • (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerPullCommand.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #12994 (See https://builds.apache.org/job/Hadoop-trunk-Commit/12994/ ) YARN-6623 . Add support to turn off launching privileged containers in (wangda: rev d3b1c6319546706c41a2011ead6c3fe208883200) (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerStopCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRmCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerRunCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRunCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test_util.cc (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/TestDockerContainerRuntime.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/test-container-executor.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommandExecutor.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.h (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/main.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerCommandExecutor.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test-string-utils.cc (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerLoadCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/configuration.h (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.h (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerInspectCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/util.c (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.h (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/resources/test/test-configurations/docker-container-executor.cfg (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerRmCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerLoadCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/string-utils.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/DockerLinuxContainerRuntime.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerStopCommand.java (edit) hadoop-yarn-project/hadoop-yarn/conf/container-executor.cfg (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/CMakeLists.txt (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/get_executable.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/common/module-configs.c (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerClient.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainers.md (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/modules/common/module-configs.h (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerPullCommand.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/DockerInspectCommand.java (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/container-executor.h (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/linux/runtime/docker/TestDockerPullCommand.java
          Hide
          leftnoteasy Wangda Tan added a comment -

          Andrew Wang, I just saw your email for branch-3.0.0-beta1 cut, is it still open for new commits? Can we merge this to branch-3.0.0-beta1?

          Show
          leftnoteasy Wangda Tan added a comment - Andrew Wang , I just saw your email for branch-3.0.0-beta1 cut, is it still open for new commits? Can we merge this to branch-3.0.0-beta1?
          Hide
          leftnoteasy Wangda Tan added a comment -

          Discussed with Eric Yang / Varun Vasudev / Shane Kumpf, Eric Yang mentioned that we can move the whitelist usability issue to a separate JIRA to unblock this one.

          So I just pushed to trunk/branch-3.0, it has some conflicts to branch-2.

          Thanks to Varun Vasudev for the hard work and thanks Miklos Szegedi / Eric Badger / Shane Kumpf / Eric Yang / Daniel Templeton / Junping Du / Andrew Wang for reviews and suggestions!

          Show
          leftnoteasy Wangda Tan added a comment - Discussed with Eric Yang / Varun Vasudev / Shane Kumpf , Eric Yang mentioned that we can move the whitelist usability issue to a separate JIRA to unblock this one. So I just pushed to trunk/branch-3.0, it has some conflicts to branch-2. Thanks to Varun Vasudev for the hard work and thanks Miklos Szegedi / Eric Badger / Shane Kumpf / Eric Yang / Daniel Templeton / Junping Du / Andrew Wang for reviews and suggestions!
          Hide
          andrew.wang Andrew Wang added a comment -

          Thanks Varun. Considering this is targeted at 2.8.2, if we're okay taking that incompatibility for a maintenance release, I'm okay taking it between beta1 and GA. I'll update the target version, though if beta1 drags on, we can try and squeeze this in.

          Show
          andrew.wang Andrew Wang added a comment - Thanks Varun. Considering this is targeted at 2.8.2, if we're okay taking that incompatibility for a maintenance release, I'm okay taking it between beta1 and GA. I'll update the target version, though if beta1 drags on, we can try and squeeze this in.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          If the decision is to ignore this bug for beta1, how about renaming the switch feature.docker.enabled to feature.docker.experimental.enabled or feature.docker.unsecure.enabled to make the decision clear to users?

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - If the decision is to ignore this bug for beta1, how about renaming the switch feature.docker.enabled to feature.docker.experimental.enabled or feature.docker.unsecure.enabled to make the decision clear to users?
          Hide
          shanekumpf@gmail.com Shane Kumpf added a comment -

          Thanks for addressing my comments. I can confirm that docker inspect is now functioning as expected. Sorry for the delay on this update, I was running into an issue with tc/network as a resource with this patch applied, however, I have been able to confirm that this is an existing issue that is unrelated to this patch. +1 (non-binding) on the latest patch.

          Show
          shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for addressing my comments. I can confirm that docker inspect is now functioning as expected. Sorry for the delay on this update, I was running into an issue with tc/network as a resource with this patch applied, however, I have been able to confirm that this is an existing issue that is unrelated to this patch. +1 (non-binding) on the latest patch.
          Hide
          vvasudev Varun Vasudev added a comment - - edited

          Andrew Wang - the one thing you should be aware of is that YARN-6623 will break backwards compatibility from earlier releases because it disables most Docker features by default. So folks upgrading from beta-1 to beta-2 will find their Docker containers will no longer work. If you're fine with that, please go ahead with beta-1.

          Show
          vvasudev Varun Vasudev added a comment - - edited Andrew Wang - the one thing you should be aware of is that YARN-6623 will break backwards compatibility from earlier releases because it disables most Docker features by default. So folks upgrading from beta-1 to beta-2 will find their Docker containers will no longer work. If you're fine with that, please go ahead with beta-1.
          Hide
          andrew.wang Andrew Wang added a comment -

          Folks, this is currently marked as the last blocker for beta1. I asked Mikos about this offline, and he explained to me that:

          • We've documented the Docker feature as experimental
          • It's off by default
          • This isn't a regression from an earlier 3.0.0 alpha release

          Given this, I'd like to drop this off the blocker list and proceed with the beta1 release.

          Alternatively, if there's some quick hack we could do to unblock the release, I'm all ears.

          Show
          andrew.wang Andrew Wang added a comment - Folks, this is currently marked as the last blocker for beta1. I asked Mikos about this offline, and he explained to me that: We've documented the Docker feature as experimental It's off by default This isn't a regression from an earlier 3.0.0 alpha release Given this, I'd like to drop this off the blocker list and proceed with the beta1 release. Alternatively, if there's some quick hack we could do to unblock the release, I'm all ears.
          Hide
          eyang Eric Yang added a comment -

          Tan, Wangda How does a non-privileged user acquire excessive permission by executing c-e? root:yarn is typically the owner of c-e binary. The user has to be root or yarn to run the binary. Hence, validation done by YARN user would be better than doing post validation after root privilege is acquired. One can argue that YARN user does not have access to check mount points, hence the validation needs to happen at root user level. If docker container is started for unprivileged user by using -u [uid]:[gid], Linux file system ACL still applies to process inside container. There will be no extra permission gain with mounting unauthorized path. In the previous implementation, there was no effective group id passed to docker. This was the reason that extra permission was gain through effective group. When this security hole is closed by YARN-4266, then there is no gain to shift validation logic to root user side for mount point permission validation.

          Show
          eyang Eric Yang added a comment - Tan, Wangda How does a non-privileged user acquire excessive permission by executing c-e? root:yarn is typically the owner of c-e binary. The user has to be root or yarn to run the binary. Hence, validation done by YARN user would be better than doing post validation after root privilege is acquired. One can argue that YARN user does not have access to check mount points, hence the validation needs to happen at root user level. If docker container is started for unprivileged user by using -u [uid] : [gid] , Linux file system ACL still applies to process inside container. There will be no extra permission gain with mounting unauthorized path. In the previous implementation, there was no effective group id passed to docker. This was the reason that extra permission was gain through effective group. When this security hole is closed by YARN-4266 , then there is no gain to shift validation logic to root user side for mount point permission validation.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Eric Yang,

          Thanks for the comments. I'm agree with #2, which we should have a way to expose allowed whitelist and other stuffs such as allowed devices.

          However, to me, limiting capability of c-e can definitely improve security. We don't want a non-privileged user acquires excessive permissions by executing c-e (excessive permissions like mount volumes, etc). Please let me know your thoughts.

          Show
          leftnoteasy Wangda Tan added a comment - Eric Yang , Thanks for the comments. I'm agree with #2, which we should have a way to expose allowed whitelist and other stuffs such as allowed devices. However, to me, limiting capability of c-e can definitely improve security. We don't want a non-privileged user acquires excessive permissions by executing c-e (excessive permissions like mount volumes, etc). Please let me know your thoughts.
          Hide
          eyang Eric Yang added a comment -

          A couple concerns:

          1. Moving code from Java to C without findbugs check for vulnerability, is risky for future enhancement.
          2. Mount point white list should be placed in visible place like common-site.xml or yarn-site.xml to let other people know about the path that can be mounted.
          3. Container-executor.cfg permission might set to 640, which prevent usability from point #2 for users.

          Container-executor binary is governed by setuid bits. A privileged user is allowed to do many things in Linux. Effort of trying to limit root user to less power, does not improve security. It only make system more difficult to service in situations that have yet been realized. Sorry that there are a lot of code been written for this JIRA. However, it seems a bit risky to push validation logic to root user side. It would have been better to reduce the scope of this JIRA to focus on disabling launching privileged containers on node manager side only in my opinion.

          The failed unit test case does not seem to be related to the latest version of patch.

          Show
          eyang Eric Yang added a comment - A couple concerns: Moving code from Java to C without findbugs check for vulnerability, is risky for future enhancement. Mount point white list should be placed in visible place like common-site.xml or yarn-site.xml to let other people know about the path that can be mounted. Container-executor.cfg permission might set to 640, which prevent usability from point #2 for users. Container-executor binary is governed by setuid bits. A privileged user is allowed to do many things in Linux. Effort of trying to limit root user to less power, does not improve security. It only make system more difficult to service in situations that have yet been realized. Sorry that there are a lot of code been written for this JIRA. However, it seems a bit risky to push validation logic to root user side. It would have been better to reduce the scope of this JIRA to focus on disabling launching privileged containers on node manager side only in my opinion. The failed unit test case does not seem to be related to the latest version of patch.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 43s Maven dependency ordering for branch
          +1 mvninstall 12m 7s trunk passed
          +1 compile 9m 11s trunk passed
          +1 checkstyle 0m 48s trunk passed
          +1 mvnsite 3m 23s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 40s trunk passed
          +1 javadoc 2m 5s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 3m 3s the patch passed
          +1 compile 5m 43s the patch passed
          +1 cc 5m 43s the patch passed
          +1 javac 5m 43s the patch passed
          +1 checkstyle 0m 48s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27)
          +1 mvnsite 3m 17s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 46s the patch passed
          +1 javadoc 1m 51s the patch passed
                Other Tests
          -1 unit 74m 59s hadoop-yarn in the patch failed.
          +1 unit 14m 35s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 0m 18s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          142m 59s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12888901/YARN-6623.013.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
          uname Linux abd271841e6d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 02e2a9b
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17622/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17622/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17622/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 12m 7s trunk passed +1 compile 9m 11s trunk passed +1 checkstyle 0m 48s trunk passed +1 mvnsite 3m 23s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 40s trunk passed +1 javadoc 2m 5s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 3m 3s the patch passed +1 compile 5m 43s the patch passed +1 cc 5m 43s the patch passed +1 javac 5m 43s the patch passed +1 checkstyle 0m 48s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27) +1 mvnsite 3m 17s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 46s the patch passed +1 javadoc 1m 51s the patch passed       Other Tests -1 unit 74m 59s hadoop-yarn in the patch failed. +1 unit 14m 35s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 18s hadoop-yarn-site in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 142m 59s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12888901/YARN-6623.013.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux abd271841e6d 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 02e2a9b Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/17622/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17622/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17622/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Looks good to me! +1 (non-binding) pending jenkins

          Show
          ebadger Eric Badger added a comment - Looks good to me! +1 (non-binding) pending jenkins
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded the wrong patch file by mistake. YARN-6623.013.patch is correct patch.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded the wrong patch file by mistake. YARN-6623 .013.patch is correct patch.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the review Eric Badger.

          Hey Varun Vasudev, thanks for the updated patch! The changes related to YARN-4266 look good. Just had one comment on the code.
          
          1850 +static int set_network(const struct configuration *command_config,
          1851 +                       const struct configuration *conf, char *out,
          1852 +                       const size_t outlen) {
          1853 +
          1854 +  int ret = 0;
          1855 +  ret = add_param_to_command_if_allowed(command_config, conf, "net",
          1856 +                                        "docker.allowed.networks", "--net=",
          1857 +                                        0, 0, out, outlen);
          1858 +  if (ret != 0) {
          1859 +    fprintf(ERRORFILE, "Could not find requested network in allowed networks\n");
          1860 +    ret = INVALID_DOCKER_NETWORK;
          1861 +  }
          1862 +  if (ret != 0) {
          1863 +    memset(out, 0, outlen);
          1864 +  }
          1865 +  return ret;
          1866 +}
          
          Something I noticed in passing. We should combine the if(ret != 0) statements here. This same thing also occurs in set_capabilities() and set_devices().
          
          

          Fixed.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the review Eric Badger . Hey Varun Vasudev, thanks for the updated patch! The changes related to YARN-4266 look good. Just had one comment on the code. 1850 +static int set_network(const struct configuration *command_config, 1851 + const struct configuration *conf, char *out, 1852 + const size_t outlen) { 1853 + 1854 + int ret = 0; 1855 + ret = add_param_to_command_if_allowed(command_config, conf, "net", 1856 + "docker.allowed.networks", "--net=", 1857 + 0, 0, out, outlen); 1858 + if (ret != 0) { 1859 + fprintf(ERRORFILE, "Could not find requested network in allowed networks\n"); 1860 + ret = INVALID_DOCKER_NETWORK; 1861 + } 1862 + if (ret != 0) { 1863 + memset(out, 0, outlen); 1864 + } 1865 + return ret; 1866 +} Something I noticed in passing. We should combine the if(ret != 0) statements here. This same thing also occurs in set_capabilities() and set_devices(). Fixed.
          Hide
          ebadger Eric Badger added a comment -

          Hey Varun Vasudev, thanks for the updated patch! The changes related to YARN-4266 look good. Just had one comment on the code.

          1850 +static int set_network(const struct configuration *command_config,
          1851 +                       const struct configuration *conf, char *out,
          1852 +                       const size_t outlen) {
          1853 +
          1854 +  int ret = 0;
          1855 +  ret = add_param_to_command_if_allowed(command_config, conf, "net",
          1856 +                                        "docker.allowed.networks", "--net=",
          1857 +                                        0, 0, out, outlen);
          1858 +  if (ret != 0) {
          1859 +    fprintf(ERRORFILE, "Could not find requested network in allowed networks\n");
          1860 +    ret = INVALID_DOCKER_NETWORK;
          1861 +  }
          1862 +  if (ret != 0) {
          1863 +    memset(out, 0, outlen);
          1864 +  }
          1865 +  return ret;
          1866 +}
          

          Something I noticed in passing. We should combine the if(ret != 0) statements here. This same thing also occurs in set_capabilities() and set_devices().

          Show
          ebadger Eric Badger added a comment - Hey Varun Vasudev , thanks for the updated patch! The changes related to YARN-4266 look good. Just had one comment on the code. 1850 +static int set_network(const struct configuration *command_config, 1851 + const struct configuration *conf, char *out, 1852 + const size_t outlen) { 1853 + 1854 + int ret = 0; 1855 + ret = add_param_to_command_if_allowed(command_config, conf, "net", 1856 + "docker.allowed.networks", "--net=", 1857 + 0, 0, out, outlen); 1858 + if (ret != 0) { 1859 + fprintf(ERRORFILE, "Could not find requested network in allowed networks\n"); 1860 + ret = INVALID_DOCKER_NETWORK; 1861 + } 1862 + if (ret != 0) { 1863 + memset(out, 0, outlen); 1864 + } 1865 + return ret; 1866 +} Something I noticed in passing. We should combine the if(ret != 0) statements here. This same thing also occurs in set_capabilities() and set_devices() .
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 13s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 50s Maven dependency ordering for branch
          +1 mvninstall 13m 49s trunk passed
          +1 compile 9m 30s trunk passed
          +1 checkstyle 0m 59s trunk passed
          +1 mvnsite 3m 47s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 44s trunk passed
          +1 javadoc 2m 14s trunk passed
                Patch Compile Tests
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 3m 22s the patch passed
          +1 compile 5m 55s the patch passed
          +1 cc 5m 55s the patch passed
          +1 javac 5m 55s the patch passed
          +1 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27)
          +1 mvnsite 3m 45s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 52s the patch passed
          +1 javadoc 2m 12s the patch passed
                Other Tests
          -1 unit 74m 52s hadoop-yarn in the patch failed.
          +1 unit 14m 18s hadoop-yarn-server-nodemanager in the patch passed.
          +1 unit 0m 16s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 29s The patch does not generate ASF License warnings.
          147m 26s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12888749/YARN-6623.012.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
          uname Linux 35de879da443 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 415e5a1
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17615/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17615/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17615/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 13s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 50s Maven dependency ordering for branch +1 mvninstall 13m 49s trunk passed +1 compile 9m 30s trunk passed +1 checkstyle 0m 59s trunk passed +1 mvnsite 3m 47s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 44s trunk passed +1 javadoc 2m 14s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 3m 22s the patch passed +1 compile 5m 55s the patch passed +1 cc 5m 55s the patch passed +1 javac 5m 55s the patch passed +1 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27) +1 mvnsite 3m 45s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 52s the patch passed +1 javadoc 2m 12s the patch passed       Other Tests -1 unit 74m 52s hadoop-yarn in the patch failed. +1 unit 14m 18s hadoop-yarn-server-nodemanager in the patch passed. +1 unit 0m 16s hadoop-yarn-site in the patch passed. +1 asflicense 0m 29s The patch does not generate ASF License warnings. 147m 26s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12888749/YARN-6623.012.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux 35de879da443 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 415e5a1 Default Java 1.8.0_144 findbugs v3.1.0-RC1 unit https://builds.apache.org/job/PreCommit-YARN-Build/17615/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17615/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17615/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded 012.patch to fix cc and unit test issues.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded 012.patch to fix cc and unit test issues.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 43s Maven dependency ordering for branch
          +1 mvninstall 13m 44s trunk passed
          +1 compile 10m 2s trunk passed
          +1 checkstyle 1m 2s trunk passed
          +1 mvnsite 4m 3s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 47s trunk passed
          +1 javadoc 2m 17s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 4m 2s the patch passed
          +1 compile 6m 30s the patch passed
          -1 cc 6m 30s hadoop-yarn-project_hadoop-yarn generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javac 6m 30s the patch passed
          +1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27)
          +1 mvnsite 3m 44s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 51s the patch passed
          +1 javadoc 2m 8s the patch passed
                Other Tests
          -1 unit 74m 48s hadoop-yarn in the patch failed.
          -1 unit 14m 29s hadoop-yarn-server-nodemanager in the patch failed.
          +1 unit 0m 24s hadoop-yarn-site in the patch passed.
          +1 asflicense 0m 44s The patch does not generate ASF License warnings.
          149m 54s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
            hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
            hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12888743/YARN-6623.011.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
          uname Linux 53320080d615 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 415e5a1
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          cc https://builds.apache.org/job/PreCommit-YARN-Build/17611/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17611/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17611/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 43s Maven dependency ordering for branch +1 mvninstall 13m 44s trunk passed +1 compile 10m 2s trunk passed +1 checkstyle 1m 2s trunk passed +1 mvnsite 4m 3s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 47s trunk passed +1 javadoc 2m 17s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 4m 2s the patch passed +1 compile 6m 30s the patch passed -1 cc 6m 30s hadoop-yarn-project_hadoop-yarn generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javac 6m 30s the patch passed +1 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27) +1 mvnsite 3m 44s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 51s the patch passed +1 javadoc 2m 8s the patch passed       Other Tests -1 unit 74m 48s hadoop-yarn in the patch failed. -1 unit 14m 29s hadoop-yarn-server-nodemanager in the patch failed. +1 unit 0m 24s hadoop-yarn-site in the patch passed. +1 asflicense 0m 44s The patch does not generate ASF License warnings. 149m 54s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime   hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12888743/YARN-6623.011.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux 53320080d615 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 415e5a1 Default Java 1.8.0_144 findbugs v3.1.0-RC1 cc https://builds.apache.org/job/PreCommit-YARN-Build/17611/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17611/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17611/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17611/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch (.011). It adds support for group-add and adds documentation for the container-executor.cfg settings.

          Eric Badger - can you please take a look at the group-add changes and confirm they're ok?

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch (.011). It adds support for group-add and adds documentation for the container-executor.cfg settings. Eric Badger - can you please take a look at the group-add changes and confirm they're ok?
          Hide
          vvasudev Varun Vasudev added a comment -

          Andrew Wang - YARN-4266 adds a new field for the Docker invocation so I need to update the patch to add it in.
          Junping Du - the discussion made sense to me. Please go ahead and file that JIRA.

          Show
          vvasudev Varun Vasudev added a comment - Andrew Wang - YARN-4266 adds a new field for the Docker invocation so I need to update the patch to add it in. Junping Du - the discussion made sense to me. Please go ahead and file that JIRA.
          Hide
          vvasudev Varun Vasudev added a comment -

          Andrew Wang - YARN-4266 adds a new field for the Docker invocation so I need to update the patch to add it in.
          Junping Du - the discussion made sense to me. Please go ahead and file that JIRA.

          Show
          vvasudev Varun Vasudev added a comment - Andrew Wang - YARN-4266 adds a new field for the Docker invocation so I need to update the patch to add it in. Junping Du - the discussion made sense to me. Please go ahead and file that JIRA.
          Hide
          djp Junping Du added a comment -

          Hi Varun Vasudev, do you agree with what we discussed in security alias thread that only fix a small (but very important) issue instead of backporting whole YARN-6623? If so, I will go ahead to file a new jira to unblock 2.8.2 release?

          Show
          djp Junping Du added a comment - Hi Varun Vasudev , do you agree with what we discussed in security alias thread that only fix a small (but very important) issue instead of backporting whole YARN-6623 ? If so, I will go ahead to file a new jira to unblock 2.8.2 release?
          Hide
          andrew.wang Andrew Wang added a comment -

          Varun, can we commit the trunk/branch-3.0 patch today? YARN-4266 is already in trunk/branch-3.0.

          Show
          andrew.wang Andrew Wang added a comment - Varun, can we commit the trunk/branch-3.0 patch today? YARN-4266 is already in trunk/branch-3.0.
          Hide
          asuresh Arun Suresh added a comment -

          Thanks for the heads up Junping Du.. Kindly do push this into branch-2 ass well

          Show
          asuresh Arun Suresh added a comment - Thanks for the heads up Junping Du .. Kindly do push this into branch-2 ass well
          Hide
          vvasudev Varun Vasudev added a comment - - edited

          Wangda Tan - please don't commit the patch. I need to incorporate YARN-4266 into this. I should have a patch for you over the weekend. Thanks!

          Show
          vvasudev Varun Vasudev added a comment - - edited Wangda Tan - please don't commit the patch. I need to incorporate YARN-4266 into this. I should have a patch for you over the weekend. Thanks!
          Hide
          djp Junping Du added a comment -

          I plan to commit the patch to trunk/branch-3.0 tomorrow if no more opposite opinions.

          I think branch-2 also need this as all docker related patch should also get landed on branch-2. CC Arun Suresh.

          Show
          djp Junping Du added a comment - I plan to commit the patch to trunk/branch-3.0 tomorrow if no more opposite opinions. I think branch-2 also need this as all docker related patch should also get landed on branch-2. CC Arun Suresh .
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Varun Vasudev and all other folks for the hard work to finish / review the patch. I just checked the latest patch, it looks fine and all new added config knobs are properly designed. So I'm +1 to the latest patch, it looks like this patch needs rebase on top of YARN-7034.

          Is there any other comments? I plan to commit the patch to trunk/branch-3.0 tomorrow if no more opposite opinions.

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Varun Vasudev and all other folks for the hard work to finish / review the patch. I just checked the latest patch, it looks fine and all new added config knobs are properly designed. So I'm +1 to the latest patch, it looks like this patch needs rebase on top of YARN-7034 . Is there any other comments? I plan to commit the patch to trunk/branch-3.0 tomorrow if no more opposite opinions.
          Hide
          ebadger Eric Badger added a comment -

          Given that docker is alpha in 2.8.2, I agree with Vinod Kumar Vavilapalli that we should not backport this to 2.8.2, as it makes significant changes to the container-executor. Nobody should even attempt seriously running docker on 2.8.2 without backporting most, if not everything that's in 2.9 and doing a large security review over their configurations and setup. So, in my mind, some docker code may be "in" 2.8.2, but it really isn't supported at all until 2.9+. The only reason I can think of to put this in 2.8.2 would be to get the changes to container-executor, but it's not obvious to me why we would need those.

          Show
          ebadger Eric Badger added a comment - Given that docker is alpha in 2.8.2, I agree with Vinod Kumar Vavilapalli that we should not backport this to 2.8.2, as it makes significant changes to the container-executor. Nobody should even attempt seriously running docker on 2.8.2 without backporting most, if not everything that's in 2.9 and doing a large security review over their configurations and setup. So, in my mind, some docker code may be "in" 2.8.2, but it really isn't supported at all until 2.9+. The only reason I can think of to put this in 2.8.2 would be to get the changes to container-executor, but it's not obvious to me why we would need those.
          Hide
          djp Junping Du added a comment -

          We have already documented that the docker feature is alpha, not for production use (and documenting more). Given this, I don't think we should add more risk to 2.8.2.

          That's also my initial thinking, but Shane Kumpf convinced me offline that this is important for 2.8.2 even as an alpha feature - indeed still alpha for 2.9 and 3.0 and seems to affect non-docker container runtime. So I change my mind to support this backport. Varun Vasudev, Shane Kumpf and Miklos Szegedi, what do you guys think?

          Show
          djp Junping Du added a comment - We have already documented that the docker feature is alpha, not for production use (and documenting more). Given this, I don't think we should add more risk to 2.8.2. That's also my initial thinking, but Shane Kumpf convinced me offline that this is important for 2.8.2 even as an alpha feature - indeed still alpha for 2.9 and 3.0 and seems to affect non-docker container runtime. So I change my mind to support this backport. Varun Vasudev , Shane Kumpf and Miklos Szegedi , what do you guys think?
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Junping Du, this is too big a change and a patch to put into 2.8.2. We have already documented that the docker feature is alpha, not for production use (and documenting more). Given this, I don't think we should add more risk to 2.8.2.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Junping Du , this is too big a change and a patch to put into 2.8.2. We have already documented that the docker feature is alpha, not for production use (and documenting more). Given this, I don't think we should add more risk to 2.8.2.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Indeed, I verified and we still need the strlen in the first case, and I do not see an overflow possibility of tmp_buffer_2, so those should be okay.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Indeed, I verified and we still need the strlen in the first case, and I do not see an overflow possibility of tmp_buffer_2, so those should be okay.
          Hide
          vvasudev Varun Vasudev added a comment -
             
               How would you detect the condition where the buffer doesn't have enough size?
          
          You copy at most bufflen-strlen(buff) characters including \0. As I said only one strlen is enough in this case.
          

          I think I understand what you're saying. The current implementation checks if the buffer has enough space to hold the final string and will return an error if there isn't enough space. Without the additional strlen, how would I check that the buffer can fit the additional string?

              381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name); 
          
              That space might need to be added to the quote_and_append_arg function for safety reasons.
          
              I didn't get this. Can you please explain?
          
          When we add a new arg then the space should be added by default in quote_and_append_arg every time.
          

          Ah I see what you're saying. Thanks for the explanation. That line of code got removed in the latest patch to address the feedback from Shane Kumpf in his review.

              Is there any benefit to strcpy + strcat?
          
          There is no need to do an strlen. Was the intention maybe to bound by the size of tmp_buffer_2?
          

          Yep. The thinking was to limit it to the size of tmp_buffer_2.

          Show
          vvasudev Varun Vasudev added a comment - How would you detect the condition where the buffer doesn't have enough size? You copy at most bufflen-strlen(buff) characters including \0. As I said only one strlen is enough in this case. I think I understand what you're saying. The current implementation checks if the buffer has enough space to hold the final string and will return an error if there isn't enough space. Without the additional strlen, how would I check that the buffer can fit the additional string? 381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name); That space might need to be added to the quote_and_append_arg function for safety reasons. I didn't get this. Can you please explain? When we add a new arg then the space should be added by default in quote_and_append_arg every time. Ah I see what you're saying. Thanks for the explanation. That line of code got removed in the latest patch to address the feedback from Shane Kumpf in his review. Is there any benefit to strcpy + strcat? There is no need to do an strlen. Was the intention maybe to bound by the size of tmp_buffer_2? Yep. The thinking was to limit it to the size of tmp_buffer_2.
          Hide
          vvasudev Varun Vasudev added a comment -

          Junping Du - makes sense to push it into 2.8.2; Andrew Wang - thanks for updating the target versions.

          Show
          vvasudev Varun Vasudev added a comment - Junping Du - makes sense to push it into 2.8.2; Andrew Wang - thanks for updating the target versions.
          Hide
          andrew.wang Andrew Wang added a comment -

          I added some target versions, might not be comprehensive.

          Show
          andrew.wang Andrew Wang added a comment - I added some target versions, might not be comprehensive.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Varun Vasudev, thank you for the updated patch. I will review it but before that let's address your questions.

          How would you detect the condition where the buffer doesn't have enough size?

          You copy at most bufflen-strlen(buff) characters including \0. As I said only one strlen is enough in this case.

          I didn't quite understand this. What would the len do? Your understanding is correct, we're checking if the left device is allowed.

          Never mind, I was just trying to replace tmp_ptr - values[i] with a length.

          381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name);

          That space might need to be added to the quote_and_append_arg function for safety reasons.

          I didn't get this. Can you please explain?

          When we add a new arg then the space should be added by default in quote_and_append_arg every time.

          Is there any benefit to strcpy + strcat?

          There is no need to do an strlen. Was the intention maybe to bound by the size of tmp_buffer_2?

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Varun Vasudev , thank you for the updated patch. I will review it but before that let's address your questions. How would you detect the condition where the buffer doesn't have enough size? You copy at most bufflen-strlen(buff) characters including \0 . As I said only one strlen is enough in this case. I didn't quite understand this. What would the len do? Your understanding is correct, we're checking if the left device is allowed. Never mind, I was just trying to replace tmp_ptr - values[i] with a length. 381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name); That space might need to be added to the quote_and_append_arg function for safety reasons. I didn't get this. Can you please explain? When we add a new arg then the space should be added by default in quote_and_append_arg every time. Is there any benefit to strcpy + strcat? There is no need to do an strlen. Was the intention maybe to bound by the size of tmp_buffer_2?
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 1m 12s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 13s Maven dependency ordering for branch
          +1 mvninstall 14m 44s trunk passed
          +1 compile 12m 26s trunk passed
          +1 checkstyle 1m 1s trunk passed
          +1 mvnsite 3m 55s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 14s trunk passed
                Patch Compile Tests
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 3m 40s the patch passed
          +1 compile 8m 20s the patch passed
          -1 cc 8m 20s hadoop-yarn-project_hadoop-yarn generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0)
          +1 javac 8m 20s the patch passed
          +1 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27)
          +1 mvnsite 3m 48s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          +1 findbugs 0m 53s the patch passed
          +1 javadoc 2m 7s the patch passed
                Other Tests
          -1 unit 84m 39s hadoop-yarn in the patch failed.
          +1 unit 15m 21s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 37s The patch does not generate ASF License warnings.
          165m 36s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands
            org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore
            org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA
            org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12888105/YARN-6623.010.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
          uname Linux cedd07b76915 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ce943eb
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17543/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          cc https://builds.apache.org/job/PreCommit-YARN-Build/17543/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17543/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17543/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17543/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 1m 12s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 13s Maven dependency ordering for branch +1 mvninstall 14m 44s trunk passed +1 compile 12m 26s trunk passed +1 checkstyle 1m 1s trunk passed +1 mvnsite 3m 55s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 14s trunk passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 3m 40s the patch passed +1 compile 8m 20s the patch passed -1 cc 8m 20s hadoop-yarn-project_hadoop-yarn generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) +1 javac 8m 20s the patch passed +1 checkstyle 1m 4s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27) +1 mvnsite 3m 48s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 53s the patch passed +1 javadoc 2m 7s the patch passed       Other Tests -1 unit 84m 39s hadoop-yarn in the patch failed. +1 unit 15m 21s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 37s The patch does not generate ASF License warnings. 165m 36s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.TestRMStoreCommands   org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore   org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA   org.apache.hadoop.yarn.server.resourcemanager.TestReservationSystemWithRMHA Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12888105/YARN-6623.010.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux cedd07b76915 3.13.0-123-generic #172-Ubuntu SMP Mon Jun 26 18:04:35 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ce943eb Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17543/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html cc https://builds.apache.org/job/PreCommit-YARN-Build/17543/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17543/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17543/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17543/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          djp Junping Du added a comment -

          Thanks Varun Vasudev for working hard on this effort and everyone for review! Shall we backport this to branch-2.8 as well, otherwise it could be a security hole for 2.8.2 as a production release?

          Show
          djp Junping Du added a comment - Thanks Varun Vasudev for working hard on this effort and everyone for review! Shall we backport this to branch-2.8 as well, otherwise it could be a security hole for 2.8.2 as a production release?
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the reviews Miklos Szegedi and Shane Kumpf!

          I've uploaded a new patch to address your comments.

          84 printWriter.println(" " + entry.getKey() + "=" + StringUtils 
          85 .join(",", entry.getValue())); 
          
          writeCommandToTempFile: entry.getKey() can still contain an = in the latest patch, which is an issue. It probably needs to be filtered in addCommandArguments.
          

          Fixed.

          701    char *get_config_path(const char *argv0) {
          702      char *executable_file = get_executable((char *) argv0);
          703      if (!executable_file) {
          704        fprintf(ERRORFILE, "realpath of executable: %s\n",
          705                errno != 0 ? strerror(errno) : "unknown");
          706        return NULL;
          707      }
          
          It is probably a good idea to check for executable_file[0] != 0 as well
          

          Fixed.

          1150      size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);
          1151      char *buffer = alloc_and_clear_memory(command_size, sizeof(char));
          1152      ret = get_docker_command(command_file, &CFG, buffer, EXECUTOR_PATH_MAX);
          
          The code passes in a different size than the actual size of the buffer.
          

          Good catch! Fixed.

          157    inline void* alloc_and_clear_memory(size_t num, size_t size) {
          158      void *ret = calloc(num, size);
          159      if (ret == NULL) {
          160        exit(OUT_OF_MEMORY);
          161      }
          162      return ret;
          163    }
          
          It might be a good idea to print an error message here.
          

          Fixed.

          42    static int add_to_buffer(char *buff, const size_t bufflen, const char *string) {
          
          Why do not you use strncat inside? It would spare one of the strlen’s.
          

          How would you detect the condition where the buffer doesn't have enough size?

          105            if(prefix != 0) {
          106              tmp_ptr = strchr(values[i], prefix);
          107              if (tmp_ptr == NULL) {
          ...
          
          This feels a little bit less readable. I would suggest having a len instead of tmp_ptr defaulted to strlen(tmp_ptr); Also, am I right that we are checking only if the left device is allowed?
          

          I didn't quite understand this. What would the len do? Your understanding is correct, we're checking if the left device is allowed.

          150      if (ret != 0) {
          151        memset(out, 0, outlen);
          152      }
          
          out[0]=0 should be sufficient, if outlen > 0 and ret != 0
          

          I prefer to memset the entire buffer.

          162 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX, strlen(CONTAINER_NAME_PREFIX))) { 
          
          There is no need of an strlen here, sizeof is sufficient and calculates compile time
          

          I tried to change this to sizeof but I got a compiler warning complaining about it.

          283 ret = add_docker_config_param(&command_config, out, outlen); 
          284 if (ret != 0) { 
          285 return BUFFER_TOO_SMALL; 
          
          Container name is not freed.
          

          Fixed.

          330 if (ret != 0) { 
          331 return BUFFER_TOO_SMALL; 
          332 } 
          
          Image name is not freed.
          

          Fixed.

          381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name); 
          
          That space might need to be added to the quote_and_append_arg function for safety reasons.
          

          I didn't get this. Can you please explain?

          564 * 2. If the path is a directory, add a '/' at the end( if not present) 
          
          There is a small typo here.
          

          Fixed.

          585 if (len <= 0) { 
          586 return NULL; 
          587 } 
          
          There is a missing free here
          

          Fixed.

          731 strncpy(tmp_buffer_2, values[i], strlen(values[i])); 
          732 strncpy(tmp_buffer_2 + strlen(values[i]), ro_suffix, strlen(ro_suffix)); 
          
          Why do you use strncpy here? Why not strcpy and strcat?
          

          Is there any benefit to strcpy + strcat?

          Can we print out the mount that was requested but failed for the read-only and read write checks? I found it a bit difficult to determine which mount was the cause of the failure. Doing the same for the devices option would be helpful as well. Something like the following:
          
                  if (permitted_rw == 0) {
                    fprintf(ERRORFILE, "Requested rw mount not found in the rw whitelist '%s', realpath=%s\n", values[i], mount_src);
                    ret = INVALID_DOCKER_RW_MOUNT;
                    goto free_and_exit;
          -snip-
               if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) {
                  fprintf(ERRORFILE, "Requested ro mount not found in the ro whitelist '%s', realpath=%s\n", values[i], mount_src);
                  ret = INVALID_DOCKER_RO_MOUNT;
                  goto free_and_exit;
                }
          

          Fixed.

          Currently only docker run works with this patch. I know you are aware of this. There are a few regressions in this patch from what was fixed in YARN-6726. It would be great if we could try to address them here. I'll list the ones I'm aware of below.
          

          Fixed.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the reviews Miklos Szegedi and Shane Kumpf ! I've uploaded a new patch to address your comments. 84 printWriter.println(" " + entry.getKey() + "=" + StringUtils 85 .join(",", entry.getValue())); writeCommandToTempFile: entry.getKey() can still contain an = in the latest patch, which is an issue. It probably needs to be filtered in addCommandArguments. Fixed. 701 char *get_config_path(const char *argv0) { 702 char *executable_file = get_executable((char *) argv0); 703 if (!executable_file) { 704 fprintf(ERRORFILE, "realpath of executable: %s\n", 705 errno != 0 ? strerror(errno) : "unknown"); 706 return NULL; 707 } It is probably a good idea to check for executable_file[0] != 0 as well Fixed. 1150 size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024); 1151 char *buffer = alloc_and_clear_memory(command_size, sizeof(char)); 1152 ret = get_docker_command(command_file, &CFG, buffer, EXECUTOR_PATH_MAX); The code passes in a different size than the actual size of the buffer. Good catch! Fixed. 157 inline void* alloc_and_clear_memory(size_t num, size_t size) { 158 void *ret = calloc(num, size); 159 if (ret == NULL) { 160 exit(OUT_OF_MEMORY); 161 } 162 return ret; 163 } It might be a good idea to print an error message here. Fixed. 42 static int add_to_buffer(char *buff, const size_t bufflen, const char *string) { Why do not you use strncat inside? It would spare one of the strlen’s. How would you detect the condition where the buffer doesn't have enough size? 105 if(prefix != 0) { 106 tmp_ptr = strchr(values[i], prefix); 107 if (tmp_ptr == NULL) { ... This feels a little bit less readable. I would suggest having a len instead of tmp_ptr defaulted to strlen(tmp_ptr); Also, am I right that we are checking only if the left device is allowed? I didn't quite understand this. What would the len do? Your understanding is correct, we're checking if the left device is allowed. 150 if (ret != 0) { 151 memset(out, 0, outlen); 152 } out[0]=0 should be sufficient, if outlen > 0 and ret != 0 I prefer to memset the entire buffer. 162 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX, strlen(CONTAINER_NAME_PREFIX))) { There is no need of an strlen here, sizeof is sufficient and calculates compile time I tried to change this to sizeof but I got a compiler warning complaining about it. 283 ret = add_docker_config_param(&command_config, out, outlen); 284 if (ret != 0) { 285 return BUFFER_TOO_SMALL; Container name is not freed. Fixed. 330 if (ret != 0) { 331 return BUFFER_TOO_SMALL; 332 } Image name is not freed. Fixed. 381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name); That space might need to be added to the quote_and_append_arg function for safety reasons. I didn't get this. Can you please explain? 564 * 2. If the path is a directory, add a '/' at the end( if not present) There is a small typo here. Fixed. 585 if (len <= 0) { 586 return NULL; 587 } There is a missing free here Fixed. 731 strncpy(tmp_buffer_2, values[i], strlen(values[i])); 732 strncpy(tmp_buffer_2 + strlen(values[i]), ro_suffix, strlen(ro_suffix)); Why do you use strncpy here? Why not strcpy and strcat? Is there any benefit to strcpy + strcat? Can we print out the mount that was requested but failed for the read-only and read write checks? I found it a bit difficult to determine which mount was the cause of the failure. Doing the same for the devices option would be helpful as well. Something like the following: if (permitted_rw == 0) { fprintf(ERRORFILE, "Requested rw mount not found in the rw whitelist '%s', realpath=%s\n", values[i], mount_src); ret = INVALID_DOCKER_RW_MOUNT; goto free_and_exit; -snip- if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) { fprintf(ERRORFILE, "Requested ro mount not found in the ro whitelist '%s', realpath=%s\n", values[i], mount_src); ret = INVALID_DOCKER_RO_MOUNT; goto free_and_exit; } Fixed. Currently only docker run works with this patch. I know you are aware of this. There are a few regressions in this patch from what was fixed in YARN-6726. It would be great if we could try to address them here. I'll list the ones I'm aware of below. Fixed.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the feedback Miklos Szegedi, Shane Kumpf. I should have a patch addressing you reviews soon.

          Miklos Szegedi

          I have some comments but before that, does the jira target 3.0-beta1?

          I would like to get it into beta1.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the feedback Miklos Szegedi , Shane Kumpf . I should have a patch addressing you reviews soon. Miklos Szegedi I have some comments but before that, does the jira target 3.0-beta1? I would like to get it into beta1.
          Hide
          shanekumpf@gmail.com Shane Kumpf added a comment -

          Thanks for the patch Varun Vasudev! Overall it looks great. Thanks for the effort. Couple of comments.

          Can we print out the mount that was requested but failed for the read-only and read write checks? I found it a bit difficult to determine which mount was the cause of the failure. Doing the same for the devices option would be helpful as well. Something like the following:

                  if (permitted_rw == 0) {
                    fprintf(ERRORFILE, "Requested rw mount not found in the rw whitelist '%s', realpath=%s\n", values[i], mount_src);
                    ret = INVALID_DOCKER_RW_MOUNT;
                    goto free_and_exit;
          -snip-
               if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) {
                  fprintf(ERRORFILE, "Requested ro mount not found in the ro whitelist '%s', realpath=%s\n", values[i], mount_src);
                  ret = INVALID_DOCKER_RO_MOUNT;
                  goto free_and_exit;
                }
          

          Currently only docker run works with this patch. I know you are aware of this. There are a few regressions in this patch from what was fixed in YARN-6726. It would be great if we could try to address them here. I'll list the ones I'm aware of below.

          1) get_docker_inspect_command uses validate_container_id directly, instead of validate_container_name.

            if (container_name == NULL || validate_container_id(container_name) != 0) {
              return INVALID_DOCKER_CONTAINER_NAME;
            }
          

          2) The --format option for docker inspect can't be quoted, or it breaks the returned output.

          quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " --format=", format);
          

          3) The container name as the final argument can't be quoted. This happens in a number of places.

          quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "", container_name);
          

          See this comment for what I had found previously.
          YARN-6726 Items

          Show
          shanekumpf@gmail.com Shane Kumpf added a comment - Thanks for the patch Varun Vasudev ! Overall it looks great. Thanks for the effort. Couple of comments. Can we print out the mount that was requested but failed for the read-only and read write checks? I found it a bit difficult to determine which mount was the cause of the failure. Doing the same for the devices option would be helpful as well. Something like the following: if (permitted_rw == 0) { fprintf(ERRORFILE, "Requested rw mount not found in the rw whitelist '%s', realpath=%s\n" , values[i], mount_src); ret = INVALID_DOCKER_RW_MOUNT; goto free_and_exit; -snip- if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) { fprintf(ERRORFILE, "Requested ro mount not found in the ro whitelist '%s', realpath=%s\n" , values[i], mount_src); ret = INVALID_DOCKER_RO_MOUNT; goto free_and_exit; } Currently only docker run works with this patch. I know you are aware of this. There are a few regressions in this patch from what was fixed in YARN-6726 . It would be great if we could try to address them here. I'll list the ones I'm aware of below. 1) get_docker_inspect_command uses validate_container_id directly, instead of validate_container_name. if (container_name == NULL || validate_container_id(container_name) != 0) { return INVALID_DOCKER_CONTAINER_NAME; } 2) The --format option for docker inspect can't be quoted, or it breaks the returned output. quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " --format=" , format); 3) The container name as the final argument can't be quoted. This happens in a number of places. quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "", container_name); See this comment for what I had found previously. YARN-6726 Items
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Thank you for the patch Varun Vasudev and for the reviews Wangda Tan and Eric Badger.
          Sorry about the delay, I had a chance to look at the latest patch now. I have some comments but before that, does the jira target 3.0-beta1?

          84 printWriter.println(" " + entry.getKey() + "=" + StringUtils 
          85 .join(",", entry.getValue())); 
          

          writeCommandToTempFile: entry.getKey() can still contain an = in the latest patch, which is an issue. It probably needs to be filtered in addCommandArguments.

          701    char *get_config_path(const char *argv0) {
          702      char *executable_file = get_executable((char *) argv0);
          703      if (!executable_file) {
          704        fprintf(ERRORFILE, "realpath of executable: %s\n",
          705                errno != 0 ? strerror(errno) : "unknown");
          706        return NULL;
          707      }
          

          It is probably a good idea to check for executable_file[0] != 0 as well

          1150      size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);
          1151      char *buffer = alloc_and_clear_memory(command_size, sizeof(char));
          1152      ret = get_docker_command(command_file, &CFG, buffer, EXECUTOR_PATH_MAX);
          

          The code passes in a different size than the actual size of the buffer.

          157    inline void* alloc_and_clear_memory(size_t num, size_t size) {
          158      void *ret = calloc(num, size);
          159      if (ret == NULL) {
          160        exit(OUT_OF_MEMORY);
          161      }
          162      return ret;
          163    }
          

          It might be a good idea to print an error message here.

          42    static int add_to_buffer(char *buff, const size_t bufflen, const char *string) {
          

          Why do not you use strncat inside? It would spare one of the strlen’s.

          105            if(prefix != 0) {
          106              tmp_ptr = strchr(values[i], prefix);
          107              if (tmp_ptr == NULL) {
          ...
          

          This feels a little bit less readable. I would suggest having a len instead of tmp_ptr defaulted to strlen(tmp_ptr); Also, am I right that we are checking only if the left device is allowed?

          150      if (ret != 0) {
          151        memset(out, 0, outlen);
          152      }
          

          out[0]=0 should be sufficient, if outlen > 0 and ret != 0

          162 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX, strlen(CONTAINER_NAME_PREFIX))) { 
          

          There is no need of an strlen here, sizeof is sufficient and calculates compile time

          283 ret = add_docker_config_param(&command_config, out, outlen); 
          284 if (ret != 0) { 
          285 return BUFFER_TOO_SMALL; 
          

          Container name is not freed.

          330 if (ret != 0) { 
          331 return BUFFER_TOO_SMALL; 
          332 } 
          

          Image name is not freed.

          381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " ", image_name); 
          

          That space might need to be added to the quote_and_append_arg function for safety reasons.

          564 * 2. If the path is a directory, add a '/' at the end( if not present) 
          

          There is a small typo here.

          585 if (len <= 0) { 
          586 return NULL; 
          587 } 
          

          There is a missing free here

          731 strncpy(tmp_buffer_2, values[i], strlen(values[i])); 
          732 strncpy(tmp_buffer_2 + strlen(values[i]), ro_suffix, strlen(ro_suffix)); 
          

          Why do you use strncpy here? Why not strcpy and strcat?

          739 memset(tmp_buffer, 0, tmp_buffer_size); 
          

          Clearing the first character should be sufficient.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Thank you for the patch Varun Vasudev and for the reviews Wangda Tan and Eric Badger . Sorry about the delay, I had a chance to look at the latest patch now. I have some comments but before that, does the jira target 3.0-beta1? 84 printWriter.println( " " + entry.getKey() + "=" + StringUtils 85 .join( "," , entry.getValue())); writeCommandToTempFile: entry.getKey() can still contain an = in the latest patch, which is an issue. It probably needs to be filtered in addCommandArguments. 701 char *get_config_path( const char *argv0) { 702 char *executable_file = get_executable(( char *) argv0); 703 if (!executable_file) { 704 fprintf(ERRORFILE, "realpath of executable: %s\n" , 705 errno != 0 ? strerror(errno) : "unknown" ); 706 return NULL; 707 } It is probably a good idea to check for executable_file[0] != 0 as well 1150 size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024); 1151 char *buffer = alloc_and_clear_memory(command_size, sizeof( char )); 1152 ret = get_docker_command(command_file, &CFG, buffer, EXECUTOR_PATH_MAX); The code passes in a different size than the actual size of the buffer. 157 inline void* alloc_and_clear_memory(size_t num, size_t size) { 158 void *ret = calloc(num, size); 159 if (ret == NULL) { 160 exit(OUT_OF_MEMORY); 161 } 162 return ret; 163 } It might be a good idea to print an error message here. 42 static int add_to_buffer( char *buff, const size_t bufflen, const char *string) { Why do not you use strncat inside? It would spare one of the strlen’s. 105 if (prefix != 0) { 106 tmp_ptr = strchr(values[i], prefix); 107 if (tmp_ptr == NULL) { ... This feels a little bit less readable. I would suggest having a len instead of tmp_ptr defaulted to strlen(tmp_ptr); Also, am I right that we are checking only if the left device is allowed? 150 if (ret != 0) { 151 memset(out, 0, outlen); 152 } out[0]=0 should be sufficient, if outlen > 0 and ret != 0 162 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX, strlen(CONTAINER_NAME_PREFIX))) { There is no need of an strlen here, sizeof is sufficient and calculates compile time 283 ret = add_docker_config_param(&command_config, out, outlen); 284 if (ret != 0) { 285 return BUFFER_TOO_SMALL; Container name is not freed. 330 if (ret != 0) { 331 return BUFFER_TOO_SMALL; 332 } Image name is not freed. 381 quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, " " , image_name); That space might need to be added to the quote_and_append_arg function for safety reasons. 564 * 2. If the path is a directory, add a '/' at the end( if not present) There is a small typo here. 585 if (len <= 0) { 586 return NULL; 587 } There is a missing free here 731 strncpy(tmp_buffer_2, values[i], strlen(values[i])); 732 strncpy(tmp_buffer_2 + strlen(values[i]), ro_suffix, strlen(ro_suffix)); Why do you use strncpy here? Why not strcpy and strcat? 739 memset(tmp_buffer, 0, tmp_buffer_size); Clearing the first character should be sufficient.
          Hide
          ebadger Eric Badger added a comment -

          I think we may want to look further into the issue with the c99 vs. c++0x compilers, but that is separate from this JIRA. I'd like to give my non-binding +1 for patch .009. Clearly we need a committer to give a binding +1, but I also think it would be good to get another pair of eyes on this since it is a large patch and because it has some potential security implications as it deals with the container-executor directly. Thanks, Varun Vasudev for all of your hard work on this!

          Show
          ebadger Eric Badger added a comment - I think we may want to look further into the issue with the c99 vs. c++0x compilers, but that is separate from this JIRA. I'd like to give my non-binding +1 for patch .009. Clearly we need a committer to give a binding +1, but I also think it would be good to get another pair of eyes on this since it is a large patch and because it has some potential security implications as it deals with the container-executor directly. Thanks, Varun Vasudev for all of your hard work on this!
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 13m 8s trunk passed
          +1 compile 8m 55s trunk passed
          +1 checkstyle 0m 54s trunk passed
          +1 mvnsite 3m 22s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 1m 49s trunk passed
                Patch Compile Tests
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 2m 59s the patch passed
          +1 compile 5m 31s the patch passed
          +1 cc 5m 31s the patch passed
          +1 javac 5m 31s the patch passed
          +1 checkstyle 0m 49s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27)
          +1 mvnsite 3m 8s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          +1 findbugs 0m 52s the patch passed
          +1 javadoc 1m 49s the patch passed
                Other Tests
          -1 unit 72m 41s hadoop-yarn in the patch failed.
          +1 unit 13m 45s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 23s The patch does not generate ASF License warnings.
          138m 50s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885822/YARN-6623.009.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
          uname Linux dcd75cc21772 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2adf8be
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17330/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17330/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17330/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17330/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 13m 8s trunk passed +1 compile 8m 55s trunk passed +1 checkstyle 0m 54s trunk passed +1 mvnsite 3m 22s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 41s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 1m 49s trunk passed       Patch Compile Tests 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 2m 59s the patch passed +1 compile 5m 31s the patch passed +1 cc 5m 31s the patch passed +1 javac 5m 31s the patch passed +1 checkstyle 0m 49s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27) +1 mvnsite 3m 8s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 52s the patch passed +1 javadoc 1m 49s the patch passed       Other Tests -1 unit 72m 41s hadoop-yarn in the patch failed. +1 unit 13m 45s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 23s The patch does not generate ASF License warnings. 138m 50s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885822/YARN-6623.009.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux dcd75cc21772 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2adf8be Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17330/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/17330/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17330/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17330/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment - - edited

          Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here.
          Weird. I've changed the initialization, should be fixed in the latest patch.

          I'm wondering why this wasn't caught by hadoopqa though. How was it that it compiled fine for hadoopqa and failed on my box? Different compliers/versions maybe? Not sure why that should matter since designated initializers don't appear to be supported in c++. My rhel VM is using gcc 4.4.7 and fails to compile. My mac, however, is using Apple LLVM version 8.1.0 and compiles fine.

          Show
          ebadger Eric Badger added a comment - - edited Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here. Weird. I've changed the initialization, should be fixed in the latest patch. I'm wondering why this wasn't caught by hadoopqa though. How was it that it compiled fine for hadoopqa and failed on my box? Different compliers/versions maybe? Not sure why that should matter since designated initializers don't appear to be supported in c++. My rhel VM is using gcc 4.4.7 and fails to compile. My mac, however, is using Apple LLVM version 8.1.0 and compiles fine.
          Hide
          vvasudev Varun Vasudev added a comment - - edited

          git apply is failing for me on DockerClient.java. It looks like it doesn't do fuzzing like patch does.

          Fixed.

          Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here.

          Weird. I've changed the initialization, should be fixed in the latest patch.

          Show
          vvasudev Varun Vasudev added a comment - - edited git apply is failing for me on DockerClient.java. It looks like it doesn't do fuzzing like patch does. Fixed. Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here. Weird. I've changed the initialization, should be fixed in the latest patch.
          Hide
          ebadger Eric Badger added a comment - - edited

          git apply is failing for me on DockerClient.java. It looks like it doesn't do fuzzing like patch does. Not sure if we need to fix that or not.

          Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here.

          [WARNING] /usr/bin/c++    -I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native -isystem /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/../../../../../hadoop-common-project/hadoop-common/src/main/native/gtest/include -I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor -I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl  -fstack-check  -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE   -o CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o -c /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc
          [WARNING] make[2]: Leaving directory `/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
          [WARNING] make[1]: Leaving directory `/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native'
          [WARNING] In file included from /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_command(const char*, const configuration*, char*, size_t)’:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:234: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:234: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_inspect_command(const char*, const configuration*, char*, size_t)’:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:264: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:264: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_load_command(const char*, const configuration*, char*, size_t)’:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:316: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:316: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_pull_command(const char*, const configuration*, char*, size_t)’:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:360: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:360: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_rm_command(const char*, const configuration*, char*, size_t)’:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:399: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:399: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_stop_command(const char*, const configuration*, char*, size_t)’:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:440: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:440: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_run_command(const char*, const configuration*, char*, size_t)’:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:814: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:814: error: expected primary-expression before ‘.’ token
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc: In member function ‘virtual void ContainerExecutor::TestDockerUtil_test_docker_config_param_Test::TestBody()’:
          [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:1057: error: expected primary-expression before ‘.’ token
          

          The error is around the struct initialization

           struct configuration command_config = {.size=0, .sections=NULL}; 
          Show
          ebadger Eric Badger added a comment - - edited git apply is failing for me on DockerClient.java. It looks like it doesn't do fuzzing like patch does. Not sure if we need to fix that or not. Additionally, the native code doesn't compile for me on my rhel 6.8 VM. It looks like it's complaining about the C99 syntax that Miklos Szegedi mentioned here . [WARNING] /usr/bin/c++ -I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src -I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native -isystem /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/../../../../../hadoop-common-project/hadoop-common/src/main/native/gtest/include -I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor -I/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl -fstack-check -g -O2 -Wall -pthread -D_FILE_OFFSET_BITS=64 -D_GNU_SOURCE -o CMakeFiles/cetest.dir/main/native/container-executor/test/utils/test_docker_util.cc.o -c /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc [WARNING] make[2]: Leaving directory `/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native' [WARNING] make[1]: Leaving directory `/home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/target/native' [WARNING] In file included from /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:24: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_command(const char*, const configuration*, char*, size_t)’: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:234: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:234: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_inspect_command(const char*, const configuration*, char*, size_t)’: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:264: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:264: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_load_command(const char*, const configuration*, char*, size_t)’: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:316: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:316: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_pull_command(const char*, const configuration*, char*, size_t)’: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:360: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:360: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_rm_command(const char*, const configuration*, char*, size_t)’: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:399: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:399: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_stop_command(const char*, const configuration*, char*, size_t)’: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:440: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:440: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c: In function ‘int get_docker_run_command(const char*, const configuration*, char*, size_t)’: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:814: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/impl/utils/docker-util.c:814: error: expected primary-expression before ‘.’ token [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc: In member function ‘virtual void ContainerExecutor::TestDockerUtil_test_docker_config_param_Test::TestBody()’: [WARNING] /home/ebadger/hadoop/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/native/container-executor/test/utils/test_docker_util.cc:1057: error: expected primary-expression before ‘.’ token The error is around the struct initialization struct configuration command_config = {.size=0, .sections=NULL};
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 22s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 26s Maven dependency ordering for branch
          +1 mvninstall 16m 39s trunk passed
          +1 compile 10m 57s trunk passed
          +1 checkstyle 1m 6s trunk passed
          +1 mvnsite 4m 18s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          -1 javadoc 0m 50s hadoop-yarn in trunk failed.
                Patch Compile Tests
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 3m 39s the patch passed
          +1 compile 5m 44s the patch passed
          +1 cc 5m 44s the patch passed
          +1 javac 5m 44s the patch passed
          +1 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27)
          +1 mvnsite 3m 29s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          +1 findbugs 0m 50s the patch passed
          -1 javadoc 0m 45s hadoop-yarn in the patch failed.
                Other Tests
          -1 unit 74m 46s hadoop-yarn in the patch failed.
          +1 unit 13m 46s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          147m 56s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:71bbb86
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885276/YARN-6623.008.patch
          Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle
          uname Linux 7003546ebe1d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / ef87d34
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17278/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17278/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn.txt
          javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17278/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17278/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17278/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 22s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 26s Maven dependency ordering for branch +1 mvninstall 16m 39s trunk passed +1 compile 10m 57s trunk passed +1 checkstyle 1m 6s trunk passed +1 mvnsite 4m 18s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. -1 javadoc 0m 50s hadoop-yarn in trunk failed.       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 3m 39s the patch passed +1 compile 5m 44s the patch passed +1 cc 5m 44s the patch passed +1 javac 5m 44s the patch passed +1 checkstyle 0m 55s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27) +1 mvnsite 3m 29s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 50s the patch passed -1 javadoc 0m 45s hadoop-yarn in the patch failed.       Other Tests -1 unit 74m 46s hadoop-yarn in the patch failed. +1 unit 13m 46s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 147m 56s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:71bbb86 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12885276/YARN-6623.008.patch Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle uname Linux 7003546ebe1d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / ef87d34 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17278/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17278/artifact/patchprocess/branch-javadoc-hadoop-yarn-project_hadoop-yarn.txt javadoc https://builds.apache.org/job/PreCommit-YARN-Build/17278/artifact/patchprocess/patch-javadoc-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17278/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17278/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17278/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -
           int is_docker_support_enabled() {
          -    return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
          -    DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg);
          +  return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
          +                            DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg)
          +         || docker_module_enabled(&CFG);
          
          The indentation here should be fixed.
          

          Fixed.

          +# The configs below deal with settings for Docker
          +#[docker]
          +#  module.enabled=## enable/disable the module. set to "true" to enable, disabled by default
          +#  docker.binary=/usr/bin/docker
          +#  allowed.capabilities=## comma seperated capabilities that can be granted, e.g CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
          +#  allowed.devices=## comma seperated list of devices that can be mounted into a container
          +#  allowed.networks=## comma seperated networks that can be used. e.g bridge,host,none
          +#  allowed.ro-mounts=## comma seperated volumes that can be mounted as read-only
          +#  allowed.rw-mounts=## comma seperate volumes that can be mounted as read-write, add the yarn local and log dirs to this list to run Hadoop jobs
          
          For the Docker configs, I think it'd be more clear if they were prefixed with docker.
          

          Fixed.

          +   <!-- /proc/mounts,/sys/fs/cgroup are always in the same place -->
             <Match>
               <Class name="org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler" />
               <Method name="parseMtab" />
               <Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" />
             </Match>
          +  <Match>
          +    <Class name="org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime" />
          +    <Method name="launchContainer" />
          +    <Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" />
          +  </Match>
          
          Since we're going to remove the hard-coded string in YARN-6968, should we leave the findbugs warning and take care of it there?
          

          Makes sense, removed.

          +  protected final void addCommandArguments(String key, String value) {
          +    if (commandArguments.containsKey(key)) {
          +      commandArguments.get(key).add(value);
          +      return;
          +    }
          
          No need to call contains() and then call get(). We can just call get and then add the value if the return value isn't null.
          

          Fixed.

          +  @Override
          +  public String toString() {
          +    StringBuffer ret = new StringBuffer("");
          +    ret.append(this.command);
          
          Any reason not to create the string with the command in the constructor? Something like
          StringBuffer ret = new StringBuffer(this.command);
          

          Fixed.

          Show
          vvasudev Varun Vasudev added a comment - int is_docker_support_enabled() { - return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY, - DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg); + return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY, + DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg) + || docker_module_enabled(&CFG); The indentation here should be fixed. Fixed. +# The configs below deal with settings for Docker +#[docker] +# module.enabled=## enable/disable the module. set to "true" to enable, disabled by default +# docker.binary=/usr/bin/docker +# allowed.capabilities=## comma seperated capabilities that can be granted, e.g CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE +# allowed.devices=## comma seperated list of devices that can be mounted into a container +# allowed.networks=## comma seperated networks that can be used. e.g bridge,host,none +# allowed.ro-mounts=## comma seperated volumes that can be mounted as read-only +# allowed.rw-mounts=## comma seperate volumes that can be mounted as read-write, add the yarn local and log dirs to this list to run Hadoop jobs For the Docker configs, I think it'd be more clear if they were prefixed with docker. Fixed. + <!-- /proc/mounts,/sys/fs/cgroup are always in the same place --> <Match> <Class name="org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler" /> <Method name="parseMtab" /> <Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" /> </Match> + <Match> + <Class name="org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime" /> + <Method name="launchContainer" /> + <Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" /> + </Match> Since we're going to remove the hard-coded string in YARN-6968, should we leave the findbugs warning and take care of it there? Makes sense, removed. + protected final void addCommandArguments(String key, String value) { + if (commandArguments.containsKey(key)) { + commandArguments.get(key).add(value); + return; + } No need to call contains() and then call get(). We can just call get and then add the value if the return value isn't null. Fixed. + @Override + public String toString() { + StringBuffer ret = new StringBuffer(""); + ret.append(this.command); Any reason not to create the string with the command in the constructor? Something like StringBuffer ret = new StringBuffer(this.command); Fixed.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 13m 43s trunk passed
          +1 compile 9m 56s trunk passed
          +1 checkstyle 0m 58s trunk passed
          +1 mvnsite 4m 2s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 0m 47s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 12s trunk passed
                Patch Compile Tests
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 3m 17s the patch passed
          +1 compile 6m 10s the patch passed
          -1 cc 6m 10s hadoop-yarn-project_hadoop-yarn generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0)
          +1 javac 6m 10s the patch passed
          +1 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27)
          +1 mvnsite 4m 16s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          +1 findbugs 0m 54s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          +1 javadoc 2m 13s the patch passed
                Other Tests
          -1 unit 72m 31s hadoop-yarn in the patch failed.
          +1 unit 13m 35s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 28s The patch does not generate ASF License warnings.
          143m 50s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884503/YARN-6623.007.patch
          Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle
          uname Linux 434ed72ce57d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / a20e710
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17208/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          cc https://builds.apache.org/job/PreCommit-YARN-Build/17208/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17208/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17208/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17208/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 13m 43s trunk passed +1 compile 9m 56s trunk passed +1 checkstyle 0m 58s trunk passed +1 mvnsite 4m 2s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 47s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 12s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 3m 17s the patch passed +1 compile 6m 10s the patch passed -1 cc 6m 10s hadoop-yarn-project_hadoop-yarn generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) +1 javac 6m 10s the patch passed +1 checkstyle 1m 0s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 23 unchanged - 4 fixed = 23 total (was 27) +1 mvnsite 4m 16s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 54s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) +1 javadoc 2m 13s the patch passed       Other Tests -1 unit 72m 31s hadoop-yarn in the patch failed. +1 unit 13m 35s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 28s The patch does not generate ASF License warnings. 143m 50s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12884503/YARN-6623.007.patch Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle uname Linux 434ed72ce57d 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / a20e710 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17208/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html cc https://builds.apache.org/job/PreCommit-YARN-Build/17208/artifact/patchprocess/diff-compile-cc-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17208/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17208/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17208/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Hey Varun Vasudev, thanks for the update. It looks like most of my first round comments were lost, so posting them again.

           int is_docker_support_enabled() {
          -    return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
          -    DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg);
          +  return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY,
          +                            DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg)
          +         || docker_module_enabled(&CFG);
          

          The indentation here should be fixed.

          +# The configs below deal with settings for Docker
          +#[docker]
          +#  module.enabled=## enable/disable the module. set to "true" to enable, disabled by default
          +#  docker.binary=/usr/bin/docker
          +#  allowed.capabilities=## comma seperated capabilities that can be granted, e.g CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE
          +#  allowed.devices=## comma seperated list of devices that can be mounted into a container
          +#  allowed.networks=## comma seperated networks that can be used. e.g bridge,host,none
          +#  allowed.ro-mounts=## comma seperated volumes that can be mounted as read-only
          +#  allowed.rw-mounts=## comma seperate volumes that can be mounted as read-write, add the yarn local and log dirs to this list to run Hadoop jobs
          

          For the Docker configs, I think it'd be more clear if they were prefixed with docker.

          +   <!-- /proc/mounts,/sys/fs/cgroup are always in the same place -->
             <Match>
               <Class name="org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler" />
               <Method name="parseMtab" />
               <Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" />
             </Match>
          +  <Match>
          +    <Class name="org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime" />
          +    <Method name="launchContainer" />
          +    <Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" />
          +  </Match>
          

          Since we're going to remove the hard-coded string in YARN-6968, should we leave the findbugs warning and take care of it there?

          +  protected final void addCommandArguments(String key, String value) {
          +    if (commandArguments.containsKey(key)) {
          +      commandArguments.get(key).add(value);
          +      return;
          +    }
          

          No need to call contains() and then call get(). We can just call get and then add the value if the return value isn't null.

          +  @Override
          +  public String toString() {
          +    StringBuffer ret = new StringBuffer("");
          +    ret.append(this.command);
          

          Any reason not to create the string with the command in the constructor? Something like
          StringBuffer ret = new StringBuffer(this.command);

          Show
          ebadger Eric Badger added a comment - Hey Varun Vasudev , thanks for the update. It looks like most of my first round comments were lost, so posting them again. int is_docker_support_enabled() { - return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY, - DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg); + return is_feature_enabled(DOCKER_SUPPORT_ENABLED_KEY, + DEFAULT_DOCKER_SUPPORT_ENABLED, &executor_cfg) + || docker_module_enabled(&CFG); The indentation here should be fixed. +# The configs below deal with settings for Docker +#[docker] +# module.enabled=## enable/disable the module. set to "true" to enable, disabled by default +# docker.binary=/usr/bin/docker +# allowed.capabilities=## comma seperated capabilities that can be granted, e.g CHOWN,DAC_OVERRIDE,FSETID,FOWNER,MKNOD,NET_RAW,SETGID,SETUID,SETFCAP,SETPCAP,NET_BIND_SERVICE,SYS_CHROOT,KILL,AUDIT_WRITE +# allowed.devices=## comma seperated list of devices that can be mounted into a container +# allowed.networks=## comma seperated networks that can be used. e.g bridge,host,none +# allowed.ro-mounts=## comma seperated volumes that can be mounted as read-only +# allowed.rw-mounts=## comma seperate volumes that can be mounted as read-write, add the yarn local and log dirs to this list to run Hadoop jobs For the Docker configs, I think it'd be more clear if they were prefixed with docker. + <!-- /proc/mounts,/sys/fs/cgroup are always in the same place --> <Match> <Class name="org.apache.hadoop.yarn.server.nodemanager.util.CgroupsLCEResourcesHandler" /> <Method name="parseMtab" /> <Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" /> </Match> + <Match> + <Class name="org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.DockerLinuxContainerRuntime" /> + <Method name="launchContainer" /> + <Bug pattern="DMI_HARDCODED_ABSOLUTE_FILENAME" /> + </Match> Since we're going to remove the hard-coded string in YARN-6968 , should we leave the findbugs warning and take care of it there? + protected final void addCommandArguments(String key, String value) { + if (commandArguments.containsKey(key)) { + commandArguments.get(key).add(value); + return; + } No need to call contains() and then call get(). We can just call get and then add the value if the return value isn't null. + @Override + public String toString() { + StringBuffer ret = new StringBuffer(""); + ret.append(this.command); Any reason not to create the string with the command in the constructor? Something like StringBuffer ret = new StringBuffer(this.command);
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the additional feedback Eric Badger and Wangda Tan!

          As an additional comment on my second pass, I don’t think need to call normalize_mounts() up front since values might not have anything in it. Also, we might have only RO or only RW mounts, so we don’t need to necessarily compute permitted_rw or permitted_ro. We only need to if we’re actually going to mount RW or RO respectively.

          Makes sense. Fixed.

          Just one quick comment: for feature enable/disable. I suggest to put under section.

          For GPU feature (YARN-6852), I made config like:

          [gpu]

          1. Enable / disable the module
            module.enabled=true/false

          If you're OK with the style, I suggest to make it consistent: You can use module-configs.h to call module_enabled and get the config.

          Fixed.

          And what is feature tc? Could you make the feature name more descriptive?

          tc is the traffic control piece for network throttling. I'll fix the documentation in a follow up patch.

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the additional feedback Eric Badger and Wangda Tan ! As an additional comment on my second pass, I don’t think need to call normalize_mounts() up front since values might not have anything in it. Also, we might have only RO or only RW mounts, so we don’t need to necessarily compute permitted_rw or permitted_ro. We only need to if we’re actually going to mount RW or RO respectively. Makes sense. Fixed. Just one quick comment: for feature enable/disable. I suggest to put under section. For GPU feature ( YARN-6852 ), I made config like: [gpu] Enable / disable the module module.enabled=true/false If you're OK with the style, I suggest to make it consistent: You can use module-configs.h to call module_enabled and get the config. Fixed. And what is feature tc? Could you make the feature name more descriptive? tc is the traffic control piece for network throttling. I'll fix the documentation in a follow up patch.
          Hide
          leftnoteasy Wangda Tan added a comment -

          Thanks Varun Vasudev,

          Apologize for the delays. I haven't done any detailed reviews yet, I will leave detailed reviews to other folks due to my availability.

          Just one quick comment: for feature enable/disable. I suggest to put under section.

          For GPU feature (YARN-6852), I made config like:

          [gpu]
          # Enable / disable the module
          module.enabled=true/false 
          

          If you're OK with the style, I suggest to make it consistent: You can use module-configs.h to call module_enabled and get the config.

          And what is feature tc? Could you make the feature name more descriptive?

          Show
          leftnoteasy Wangda Tan added a comment - Thanks Varun Vasudev , Apologize for the delays. I haven't done any detailed reviews yet, I will leave detailed reviews to other folks due to my availability. Just one quick comment: for feature enable/disable. I suggest to put under section. For GPU feature ( YARN-6852 ), I made config like: [gpu] # Enable / disable the module module.enabled= true / false If you're OK with the style, I suggest to make it consistent: You can use module-configs.h to call module_enabled and get the config. And what is feature tc ? Could you make the feature name more descriptive?
          Hide
          ebadger Eric Badger added a comment -
          container-executor.c:run_docker()
          
          +  docker_command = construct_docker_command(command_file);
          +  docker_binary = get_docker_binary(&CFG);
          
          I don’t see these getting freed. Am I missing this invocation somewhere?
          container-executor.c:run_docker()
          
          We call the execvp function, the running program will get replaced by the docker invocation.
          

          Ahhh yes of course. Disregard my comment

          if (command == NULL || (strcmp(command, docker_command) != 0)) {
            ret = INCORRECT_COMMAND;
          }
          
          Is command guaranteed to be null-terminated? It comes from the configuration file, which is a Java creation and I don’t think Java null-terminates. This comment is probably relevant for quite a few other places that are doing string operations. We need to be very safe about this or else we risk possibly overrunning into random regions of the heap. A safe alternative would be to use the “n” version of all the string operations. This patch uses a mixed bag of the regular versions and their accompanying “n” versions. I don’t quite understand the reasoning behind the usage of each. If we guarantee that the string is null terminated (and always null terminated) then we don’t need the “n” versions. But even if we guarantee that the input string is null terminated, it may accidentally have the null character chopped off by an off by 1 error in a strdup or something like that. So my preference here would be to use the “n” versions of all of the string functions. Thoughts?
          command is guaranteed to be null terminated by the confiugration functions. If we use the 'n' versions of the function we end up doing a 'begins with' match instead of an exact match which could cause problems(e.g. "inspect" would match "inspectcommand")
          

          Yea I guess that makes sense. I still think we should be consistent on the usage of the str vs strn functions, though. There are still places in the patch using both, seemingly interchangeably.

          docker-util.c: get_docker_load_command()
          
          +  image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config);
          +  if (image_name == NULL) {
          +    return INVALID_DOCKER_IMAGE_NAME;
          +  }
          +
          +  memset(out, 0, outlen);
          
          Maybe being paranoid is a good thing, but get_docker_load_command and any of the other commands from get_docker_command are always passed in a buffer that has already been nulled out, so the memset here is redundant. I imagine it will be similar in the other get_*_comand functions. Maybe we should put a comment saying that it is the caller’s responsibility to null out the buffer that they pass in and get rid of these memsets? Either that or keep the memset here and get rid of the resets higher up in get_docker_command?
          I've gotten rid of the memset in the higher calls and left the memset in the get_docker_* functions. I just feel more comfortable having the memset there.
          

          That sounds good to me

          docker-util.c: get_docker_load_command()
          
          +  image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config);
          +  if (image_name == NULL) {
          +    return INVALID_DOCKER_IMAGE_NAME;
          +  }
          
          We should validate the image name here as we do in get_docker_pull_command below.
          You can pass tar files to docker load so we can't do much validation here. Docker save lets you save the image to any file you want and then you load it using docker load.
          

          Ah that’s true, good point.

          docker-util.c
          
          +static int add_param_to_command(const struct configuration *command_config, const char *key, const char *param,
          +                                 const int with_argument, char *out, const size_t outlen) {
          +  size_t tmp_buffer_size = 1024;
          
          What’s the reasoning behind the buffer size here? It’s hardcoded to 1024 in a bunch of places. This doesn’t follow the system path limit, the EXECUTOR_PATH_MAX or the system argument length.
          It's just a temporary buffer. The size doesn't matter too much because quote_and_append_arg will resize the buffer if it's too small.
          

          That makes sense. Shouldn’t we bound the size that quote_and_append() can realloc() to, though? I think the proper size would be MIN(_SC_ARG_MAX, 128*1024)

          docker-util.c:set_network()
          
          +  } else if (value == NULL) {
          +    ret = 0;
          +    goto free_and_exit;
          
          Since we didn’t find any network in the configuration, we didn’t set a network. That seems like it should print out an error and/or fail. Right now it will just silently do nothing.
          You can launch docker containers with no network specified. It'll just pick up the default network.
          

          Ah ok. No problem then

          docker-util.c:check_mount_permitted()
          
          +    // directory check
          +    permitted_mount_len = strlen(permitted_mounts[i]);
          +    if (permitted_mount_len > 0
          +        && permitted_mounts[i][permitted_mount_len - 1] == '/') {
          +      if (strncmp(requested, permitted_mounts[i], permitted_mount_len) == 0) {
          +        return 1;
          
          This looks potentially dangerous. Though we are normalizing the paths in all of the current invocations, someone in the future may add in another invocation that doesn’t normalize. I think at a bare minimum we need a comment saying that this function requires all symlinks to be resolved in the path that is passed to this function. Because if we allow symlinks to be followed then we will get different behavior because of how docker treats symlinks. A possible fix for this could be to combine get_src_mount() and get_real_src_mount() into a single function that parses the mount string to get the source and then resolves it.
          I've moved the normalize_mount call into the check_mount_permitted function. Does that address your concern?
          

          Yes, this should fix it since we’re checking exactly what we will be mounting against exactly what is allowed

          docker-util.c:set_capabilities()
          
          +  if (values != NULL) {
          +    if(permitted_values != NULL) {
          +      for (i = 0; values[i] != NULL; ++i) {
          +        memset(tmp_buffer, 0, tmp_buffer_size);
          +        permitted = 0;
          +        for (j = 0; permitted_values[j] != NULL; ++j) {
          +          if (strcmp(values[i], permitted_values[j]) == 0) {
          +            permitted = 1;
          +            break;
          +          }
          +        }
          
          I think this is the 3rd time I’ve seen this structure. I think it would be nice if we had a single function that took in a double array for permitted_values, and the value to be checked to see whether that value is permitted. That way we don’t have to duplicate all of the looping and can minimize the amount of code that we need to maintain. We could use this for networks, capabilities, mounts, devices, etc.
          Fixed. I added a new function called add_param_to_command_if_allowed which if used for network, capabilities and devices. Mounts has it's own function due to the read-write behaviour.
          

          Perfect, this will compact the code quite a bit.

          As a general comment for docker-util.c, it would be nice if we could consolidate some of the code in the get_command functions and in the set functions. There is a lot of redundant code in there. I’m worried that someone in the future will change part of the flow in one function (possibly a bug fix) that will then not get propagated to the rest of the similar code. This wouldn't necessarily have to be a part of this patch, but it might get kicked down the line and never get in if we don't do it now.
          I think I've addressed some of this as part of the latest patch. Can you take a look and let me know?
          

          I think you’ve done a good job consolidating. Especially with the permitted values. The functions are much leaner now.

          Thank you the comprehensive review. Very much appreciated.
          

          My pleasure!

          +static int add_mounts(const struct configuration *command_config, const struct configuration *conf, const char *key,
          +                      const int ro, char *out, const size_t outlen) {
          +  size_t tmp_buffer_size = 1024;
          +  const char *ro_suffix = "";
          +  const char *tmp_path_buffer[2] = {NULL, NULL};
          +  char *tmp_buffer = (char *) alloc_and_clear_memory(tmp_buffer_size, sizeof(char));
          +  char **permitted_ro_mounts = get_configuration_values_delimiter("allowed.ro-mounts",
          +                                                                  CONTAINER_EXECUTOR_CFG_DOCKER_SECTION, conf, ",");
          +  char **permitted_rw_mounts = get_configuration_values_delimiter("allowed.rw-mounts",
          +                                                                  CONTAINER_EXECUTOR_CFG_DOCKER_SECTION, conf, ",");
          +  char **values = get_configuration_values_delimiter(key, DOCKER_COMMAND_FILE_SECTION, command_config, ",");
          +  char *tmp_buffer_2 = NULL, *mount_src = NULL;
          +  const char *container_executor_cfg_path = normalize_mount(get_config_path(""));
          +  int i = 0, permitted_rw = 0, permitted_ro = 0, ret = 0;
          +  if (ro != 0) {
          +    ro_suffix = ":ro";
          +  }
          +
          +  ret = normalize_mounts(permitted_ro_mounts);
          +  ret |= normalize_mounts(permitted_rw_mounts);
          +  if (ret != 0) {
          +    fprintf(ERRORFILE, "Unable to find permitted docker mounts on disk\n");
          +    ret = MOUNT_ACCESS_ERROR;
          +    goto free_and_exit;
          +  }
          +
          +  if (values != NULL) {
          +    for (i = 0; values[i] != NULL; ++i) {
          +      mount_src = get_mount_source(values[i]);
          +      if (mount_src == NULL) {
          +        ret = INVALID_DOCKER_MOUNT;
          +        goto free_and_exit;
          +      }
          +      permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts, mount_src);
          +      permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts, mount_src);
          +      if (permitted_ro == -1 || permitted_rw == -1) {
          +        ret = INVALID_DOCKER_MOUNT;
          +        goto free_and_exit;
          +      }
          +      // rw mount
          +      if (ro == 0) {
          +        if (permitted_rw == 0) {
          +          fprintf(ERRORFILE, "Invalid docker rw mount '%s', realpath=%s\n", values[i], mount_src);
          +          ret = INVALID_DOCKER_RW_MOUNT;
          +          goto free_and_exit;
          +        } else {
          +          // determine if the user can modify the container-executor.cfg file
          +          tmp_path_buffer[0] = normalize_mount(mount_src);
          +          // just re-use the function, flip the args to check if the container-executor path is in the requested
          +          // mount point
          +          ret = check_mount_permitted(tmp_path_buffer, container_executor_cfg_path);
          +          free((void *) tmp_path_buffer[0]);
          +          if (ret == 1) {
          +            fprintf(ERRORFILE, "Attempting to mount a parent directory '%s' of container-executor.cfg as read-write\n",
          +                    values[i]);
          +            ret = INVALID_DOCKER_RW_MOUNT;
          +            goto free_and_exit;
          +          }
          +        }
          +      }
          +      //ro mount
          +      if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) {
          +        ret = INVALID_DOCKER_RO_MOUNT;
          +        goto free_and_exit;
          +      }
          +      tmp_buffer_2 = (char *) alloc_and_clear_memory(strlen(values[i]) + strlen(ro_suffix) + 1, sizeof(char));
          +      strncpy(tmp_buffer_2, values[i], strlen(values[i]));
          +      strncpy(tmp_buffer_2 + strlen(values[i]), ro_suffix, strlen(ro_suffix));
          +      quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "-v ", tmp_buffer_2);
          +      ret = add_to_buffer(out, outlen, tmp_buffer);
          +      free(tmp_buffer_2);
          +      free(mount_src);
          +      tmp_buffer_2 = NULL;
          +      mount_src = NULL;
          +      memset(tmp_buffer, 0, tmp_buffer_size);
          +      if (ret != 0) {
          +        ret = BUFFER_TOO_SMALL;
          +        goto free_and_exit;
          +      }
          +    }
          +  }
          +
          +  free_and_exit:
          +  free_values(permitted_ro_mounts);
          +  free_values(permitted_rw_mounts);
          +  free_values(values);
          +  free(mount_src);
          +  free((void *) container_executor_cfg_path);
          +  free(tmp_buffer);
          +  if (ret != 0) {
          +    memset(out, 0, outlen);
          +  }
          +  return ret;
          +}
          

          As an additional comment on my second pass, I don’t think need to call normalize_mounts() up front since values might not have anything in it. Also, we might have only RO or only RW mounts, so we don’t need to necessarily compute permitted_rw or permitted_ro. We only need to if we’re actually going to mount RW or RO respectively.

          The rest of your responses that I didn't touch on looked like they were fixed correctly, so I left them out of this comment for brevity.

          Show
          ebadger Eric Badger added a comment - container-executor.c:run_docker() + docker_command = construct_docker_command(command_file); + docker_binary = get_docker_binary(&CFG); I don’t see these getting freed. Am I missing this invocation somewhere? container-executor.c:run_docker() We call the execvp function, the running program will get replaced by the docker invocation. Ahhh yes of course. Disregard my comment if (command == NULL || (strcmp(command, docker_command) != 0)) { ret = INCORRECT_COMMAND; } Is command guaranteed to be null-terminated? It comes from the configuration file, which is a Java creation and I don’t think Java null-terminates. This comment is probably relevant for quite a few other places that are doing string operations. We need to be very safe about this or else we risk possibly overrunning into random regions of the heap. A safe alternative would be to use the “n” version of all the string operations. This patch uses a mixed bag of the regular versions and their accompanying “n” versions. I don’t quite understand the reasoning behind the usage of each. If we guarantee that the string is null terminated (and always null terminated) then we don’t need the “n” versions. But even if we guarantee that the input string is null terminated, it may accidentally have the null character chopped off by an off by 1 error in a strdup or something like that. So my preference here would be to use the “n” versions of all of the string functions. Thoughts? command is guaranteed to be null terminated by the confiugration functions. If we use the 'n' versions of the function we end up doing a 'begins with' match instead of an exact match which could cause problems(e.g. "inspect" would match "inspectcommand") Yea I guess that makes sense. I still think we should be consistent on the usage of the str vs strn functions, though. There are still places in the patch using both, seemingly interchangeably. docker-util.c: get_docker_load_command() + image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config); + if (image_name == NULL) { + return INVALID_DOCKER_IMAGE_NAME; + } + + memset(out, 0, outlen); Maybe being paranoid is a good thing, but get_docker_load_command and any of the other commands from get_docker_command are always passed in a buffer that has already been nulled out, so the memset here is redundant. I imagine it will be similar in the other get_*_comand functions. Maybe we should put a comment saying that it is the caller’s responsibility to null out the buffer that they pass in and get rid of these memsets? Either that or keep the memset here and get rid of the resets higher up in get_docker_command? I've gotten rid of the memset in the higher calls and left the memset in the get_docker_* functions. I just feel more comfortable having the memset there. That sounds good to me docker-util.c: get_docker_load_command() + image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config); + if (image_name == NULL) { + return INVALID_DOCKER_IMAGE_NAME; + } We should validate the image name here as we do in get_docker_pull_command below. You can pass tar files to docker load so we can't do much validation here. Docker save lets you save the image to any file you want and then you load it using docker load. Ah that’s true, good point. docker-util.c +static int add_param_to_command(const struct configuration *command_config, const char *key, const char *param, + const int with_argument, char *out, const size_t outlen) { + size_t tmp_buffer_size = 1024; What’s the reasoning behind the buffer size here? It’s hardcoded to 1024 in a bunch of places. This doesn’t follow the system path limit, the EXECUTOR_PATH_MAX or the system argument length. It's just a temporary buffer. The size doesn't matter too much because quote_and_append_arg will resize the buffer if it's too small. That makes sense. Shouldn’t we bound the size that quote_and_append() can realloc() to, though? I think the proper size would be MIN(_SC_ARG_MAX, 128*1024) docker-util.c:set_network() + } else if (value == NULL) { + ret = 0; + goto free_and_exit; Since we didn’t find any network in the configuration, we didn’t set a network. That seems like it should print out an error and/or fail. Right now it will just silently do nothing. You can launch docker containers with no network specified. It'll just pick up the default network. Ah ok. No problem then docker-util.c:check_mount_permitted() + // directory check + permitted_mount_len = strlen(permitted_mounts[i]); + if (permitted_mount_len > 0 + && permitted_mounts[i][permitted_mount_len - 1] == '/') { + if (strncmp(requested, permitted_mounts[i], permitted_mount_len) == 0) { + return 1; This looks potentially dangerous. Though we are normalizing the paths in all of the current invocations, someone in the future may add in another invocation that doesn’t normalize. I think at a bare minimum we need a comment saying that this function requires all symlinks to be resolved in the path that is passed to this function. Because if we allow symlinks to be followed then we will get different behavior because of how docker treats symlinks. A possible fix for this could be to combine get_src_mount() and get_real_src_mount() into a single function that parses the mount string to get the source and then resolves it. I've moved the normalize_mount call into the check_mount_permitted function. Does that address your concern? Yes, this should fix it since we’re checking exactly what we will be mounting against exactly what is allowed docker-util.c:set_capabilities() + if (values != NULL) { + if(permitted_values != NULL) { + for (i = 0; values[i] != NULL; ++i) { + memset(tmp_buffer, 0, tmp_buffer_size); + permitted = 0; + for (j = 0; permitted_values[j] != NULL; ++j) { + if (strcmp(values[i], permitted_values[j]) == 0) { + permitted = 1; + break; + } + } I think this is the 3rd time I’ve seen this structure. I think it would be nice if we had a single function that took in a double array for permitted_values, and the value to be checked to see whether that value is permitted. That way we don’t have to duplicate all of the looping and can minimize the amount of code that we need to maintain. We could use this for networks, capabilities, mounts, devices, etc. Fixed. I added a new function called add_param_to_command_if_allowed which if used for network, capabilities and devices. Mounts has it's own function due to the read-write behaviour. Perfect, this will compact the code quite a bit. As a general comment for docker-util.c, it would be nice if we could consolidate some of the code in the get_command functions and in the set functions. There is a lot of redundant code in there. I’m worried that someone in the future will change part of the flow in one function (possibly a bug fix) that will then not get propagated to the rest of the similar code. This wouldn't necessarily have to be a part of this patch, but it might get kicked down the line and never get in if we don't do it now. I think I've addressed some of this as part of the latest patch. Can you take a look and let me know? I think you’ve done a good job consolidating. Especially with the permitted values. The functions are much leaner now. Thank you the comprehensive review. Very much appreciated. My pleasure! +static int add_mounts(const struct configuration *command_config, const struct configuration *conf, const char *key, + const int ro, char *out, const size_t outlen) { + size_t tmp_buffer_size = 1024; + const char *ro_suffix = ""; + const char *tmp_path_buffer[2] = {NULL, NULL}; + char *tmp_buffer = (char *) alloc_and_clear_memory(tmp_buffer_size, sizeof(char)); + char **permitted_ro_mounts = get_configuration_values_delimiter("allowed.ro-mounts", + CONTAINER_EXECUTOR_CFG_DOCKER_SECTION, conf, ","); + char **permitted_rw_mounts = get_configuration_values_delimiter("allowed.rw-mounts", + CONTAINER_EXECUTOR_CFG_DOCKER_SECTION, conf, ","); + char **values = get_configuration_values_delimiter(key, DOCKER_COMMAND_FILE_SECTION, command_config, ","); + char *tmp_buffer_2 = NULL, *mount_src = NULL; + const char *container_executor_cfg_path = normalize_mount(get_config_path("")); + int i = 0, permitted_rw = 0, permitted_ro = 0, ret = 0; + if (ro != 0) { + ro_suffix = ":ro"; + } + + ret = normalize_mounts(permitted_ro_mounts); + ret |= normalize_mounts(permitted_rw_mounts); + if (ret != 0) { + fprintf(ERRORFILE, "Unable to find permitted docker mounts on disk\n"); + ret = MOUNT_ACCESS_ERROR; + goto free_and_exit; + } + + if (values != NULL) { + for (i = 0; values[i] != NULL; ++i) { + mount_src = get_mount_source(values[i]); + if (mount_src == NULL) { + ret = INVALID_DOCKER_MOUNT; + goto free_and_exit; + } + permitted_rw = check_mount_permitted((const char **) permitted_rw_mounts, mount_src); + permitted_ro = check_mount_permitted((const char **) permitted_ro_mounts, mount_src); + if (permitted_ro == -1 || permitted_rw == -1) { + ret = INVALID_DOCKER_MOUNT; + goto free_and_exit; + } + // rw mount + if (ro == 0) { + if (permitted_rw == 0) { + fprintf(ERRORFILE, "Invalid docker rw mount '%s', realpath=%s\n", values[i], mount_src); + ret = INVALID_DOCKER_RW_MOUNT; + goto free_and_exit; + } else { + // determine if the user can modify the container-executor.cfg file + tmp_path_buffer[0] = normalize_mount(mount_src); + // just re-use the function, flip the args to check if the container-executor path is in the requested + // mount point + ret = check_mount_permitted(tmp_path_buffer, container_executor_cfg_path); + free((void *) tmp_path_buffer[0]); + if (ret == 1) { + fprintf(ERRORFILE, "Attempting to mount a parent directory '%s' of container-executor.cfg as read-write\n", + values[i]); + ret = INVALID_DOCKER_RW_MOUNT; + goto free_and_exit; + } + } + } + //ro mount + if (ro != 0 && permitted_ro == 0 && permitted_rw == 0) { + ret = INVALID_DOCKER_RO_MOUNT; + goto free_and_exit; + } + tmp_buffer_2 = (char *) alloc_and_clear_memory(strlen(values[i]) + strlen(ro_suffix) + 1, sizeof(char)); + strncpy(tmp_buffer_2, values[i], strlen(values[i])); + strncpy(tmp_buffer_2 + strlen(values[i]), ro_suffix, strlen(ro_suffix)); + quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "-v ", tmp_buffer_2); + ret = add_to_buffer(out, outlen, tmp_buffer); + free(tmp_buffer_2); + free(mount_src); + tmp_buffer_2 = NULL; + mount_src = NULL; + memset(tmp_buffer, 0, tmp_buffer_size); + if (ret != 0) { + ret = BUFFER_TOO_SMALL; + goto free_and_exit; + } + } + } + + free_and_exit: + free_values(permitted_ro_mounts); + free_values(permitted_rw_mounts); + free_values(values); + free(mount_src); + free((void *) container_executor_cfg_path); + free(tmp_buffer); + if (ret != 0) { + memset(out, 0, outlen); + } + return ret; +} As an additional comment on my second pass, I don’t think need to call normalize_mounts() up front since values might not have anything in it. Also, we might have only RO or only RW mounts, so we don’t need to necessarily compute permitted_rw or permitted_ro . We only need to if we’re actually going to mount RW or RO respectively. The rest of your responses that I didn't touch on looked like they were fixed correctly, so I left them out of this comment for brevity.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for branch
          +1 mvninstall 14m 17s trunk passed
          +1 compile 9m 49s trunk passed
          +1 checkstyle 1m 5s trunk passed
          +1 mvnsite 4m 33s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 13s trunk passed
                Patch Compile Tests
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 3m 18s the patch passed
          +1 compile 6m 6s the patch passed
          +1 cc 6m 6s the patch passed
          +1 javac 6m 6s the patch passed
          -0 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 23 new + 23 unchanged - 4 fixed = 46 total (was 27)
          +1 mvnsite 3m 53s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          +1 findbugs 0m 56s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          +1 javadoc 2m 8s the patch passed
                Other Tests
          -1 unit 80m 58s hadoop-yarn in the patch failed.
          -1 unit 14m 3s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 26s The patch does not generate ASF License warnings.
          153m 26s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
            hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer
            TEST-cetest
            TEST-cetest



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883581/YARN-6623.006.patch
          Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle
          uname Linux 11b5d915f7d3 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 8196a07
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17119/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17119/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17119/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17119/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17119/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17119/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 10s Maven dependency ordering for branch +1 mvninstall 14m 17s trunk passed +1 compile 9m 49s trunk passed +1 checkstyle 1m 5s trunk passed +1 mvnsite 4m 33s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 13s trunk passed       Patch Compile Tests 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 3m 18s the patch passed +1 compile 6m 6s the patch passed +1 cc 6m 6s the patch passed +1 javac 6m 6s the patch passed -0 checkstyle 0m 57s hadoop-yarn-project/hadoop-yarn: The patch generated 23 new + 23 unchanged - 4 fixed = 46 total (was 27) +1 mvnsite 3m 53s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 56s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) +1 javadoc 2m 8s the patch passed       Other Tests -1 unit 80m 58s hadoop-yarn in the patch failed. -1 unit 14m 3s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 26s The patch does not generate ASF License warnings. 153m 26s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer   TEST-cetest   TEST-cetest Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12883581/YARN-6623.006.patch Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle uname Linux 11b5d915f7d3 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 8196a07 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17119/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17119/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17119/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17119/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17119/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17119/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thank you for the review Eric Badger!


          Should we fix the no newline at the end of file warnings? The apply tool complains about them.

          Fixed.


          DockerCommand:getCommandArguments()
          
          public Map<String, List<String>> getCommandArguments() {
            return Collections.unmodifiableMap(commandArguments);
          }
          
          This will return the command as well as the arguments. Unless we are considering the /usr/docker to be the actual command and inspect to be one of the arguments. If that’s what we’re expecting to happen here, then the name is a little bit misleading. This might be more of a problem with how commandArguments is named than how this function is named.
          

          Renamed function to getCommandWithArguments. Is that ok?


          container-executor.c:construct_docker_command()
          
          +char *construct_docker_command(const char *command_file) {
          +  int ret = 0;
          +  char *buffer = (char *) malloc(EXECUTOR_PATH_MAX * sizeof(char));
          
          This should use _SC_ARG_MAX as we did in YARN-6988
          size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);
          Also, why not use calloc() instead of malloc() and then memset()?
          container-executor.c:run_docker()
          

          Fixed; changed to use the alloc_and_clear function.


          +  docker_command = construct_docker_command(command_file);
          +  docker_binary = get_docker_binary(&CFG);
          
          I don’t see these getting freed. Am I missing this invocation somewhere?
          container-executor.c:run_docker()
          
          

          We call the execvp function, the running program will get replaced by the docker invocation.


          +  memset(docker_command_with_binary, 0, EXECUTOR_PATH_MAX);
          
          Is this necessary? We allocate the memory with calloc() which already clears all of the memory upon allocation.
          

          Yep. Fixed.


          {
          container-executor.h
          
          // get the executable's filename
          char* get_executable(char *argv0);
          
          Do we need this declaration (in container-executor.h) since we have moved that declaration into get_executable.h? We should also add get_executable.h in the appropriate places. Looks like main.c and test-container-executor.c both call get_executable.
          

          You're correct; fixed


          main.c:assert_valid_setup()
          
          -    fprintf(ERRORFILE,"realpath of executable: %s\n",
          -      errno != 0 ? strerror(errno) : "unknown");
          -    flush_and_close_log_files();
          -    exit(-1);
          +    fprintf(ERRORFILE, "realpath of executable: %s\n",
          +            errno != 0 ? strerror(errno) : "unknown");
          +    exit(INVALID_CONFIG_FILE);
          
          Why don’t we want to flush the log files anymore?
          

          Fixed.


          util.c:alloc_and_clear_memory()
          
          +void* alloc_and_clear_memory(size_t num, size_t size) {
          +  void *ret = calloc(num, size);
          +  if (ret == NULL) {
          +    exit(OUT_OF_MEMORY);
          +  }
          +  return ret;
          +}
          
          Should we inline this? Also, if we’re going to use this function, then all calloc calls should be replaced with this (like the ones I mentioned above)
          util.h
          

          Fixed(made function inline and replaced calloc invocations with alloc_and_clear)


          // DOCKER_CONTAINER_NAME_INVALID = 41,
          
          Should add (NOT USED) to follow convention
          docker-util.c:read_and_verify_command_file()
          
          

          Fixed.


          if (command == NULL || (strcmp(command, docker_command) != 0)) {
            ret = INCORRECT_COMMAND;
          }
          
          Is command guaranteed to be null-terminated? It comes from the configuration file, which is a Java creation and I don’t think Java null-terminates. This comment is probably relevant for quite a few other places that are doing string operations. We need to be very safe about this or else we risk possibly overrunning into random regions of the heap. A safe alternative would be to use the “n” version of all the string operations. This patch uses a mixed bag of the regular versions and their accompanying “n” versions. I don’t quite understand the reasoning behind the usage of each. If we guarantee that the string is null terminated (and always null terminated) then we don’t need the “n” versions. But even if we guarantee that the input string is null terminated, it may accidentally have the null character chopped off by an off by 1 error in a strdup or something like that. So my preference here would be to use the “n” versions of all of the string functions. Thoughts?
          

          command is guaranteed to be null terminated by the confiugration functions. If we use the 'n' versions of the function we end up doing a 'begins with' match instead of an exact match which could cause problems(e.g. "inspect" would match "inspectcommand")


          docker-util.c:read_and_verify_command_file()
          
          +  if (current_len + string_len < bufflen - 1) {
          +    strncpy(buff + current_len, string, bufflen - current_len);
          +    buff[current_len + string_len] = '\0';
          +    return 0;
          +  }
          
          Here it is copying over bufflen - current_len bytes, when we really only have string_len bytes to copy. It’s not going to overflow buff, but we might copy some extra garbage past the end of string if it’s not null terminated.
          

          Good point, fixed.


          docker-util.c: add_docker_config_param()
          
          +static int add_docker_config_param(const struct configuration *command_config, char *out, const size_t outlen) {
          +  int ret = 0;
          +  size_t tmp_buffer_size = 4096;
          +  char *tmp_buffer;
          +  char *config = get_configuration_value("docker-config", DOCKER_COMMAND_FILE_SECTION, command_config);
          +  if (config != NULL) {
          +    tmp_buffer = (char *) alloc_and_clear_memory(tmp_buffer_size, sizeof(char));
          +    quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "--config=", config);
          +    ret = add_to_buffer(out, outlen, tmp_buffer);
          +    free(tmp_buffer);
          +    free(config);
          +  }
          +  return ret;
          
          Is this function necessary? Can’t it be done in a call to add_param_to_command?
          

          You're correct again . Fixed.


          docker-util.c: get_docker_load_command()
          
          +  image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config);
          +  if (image_name == NULL) {
          +    return INVALID_DOCKER_IMAGE_NAME;
          +  }
          +
          +  memset(out, 0, outlen);
          
          Maybe being paranoid is a good thing, but get_docker_load_command and any of the other commands from get_docker_command are always passed in a buffer that has already been nulled out, so the memset here is redundant. I imagine it will be similar in the other get_*_comand functions. Maybe we should put a comment saying that it is the caller’s responsibility to null out the buffer that they pass in and get rid of these memsets? Either that or keep the memset here and get rid of the resets higher up in get_docker_command?
          

          I've gotten rid of the memset in the higher calls and left the memset in the get_docker_* functions. I just feel more comfortable having the memset there.


          docker-util.c: get_docker_load_command()
          
          +  image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config);
          +  if (image_name == NULL) {
          +    return INVALID_DOCKER_IMAGE_NAME;
          +  }
          
          We should validate the image name here as we do in get_docker_pull_command below.
          

          You can pass tar files to docker load so we can't do much validation here. Docker save lets you save the image to any file you want and then you load it using docker load.


          docker-util.c: get_docker_rm_command()
          
          +  container_name = get_configuration_value("name", DOCKER_COMMAND_FILE_SECTION, &command_config);
          +  if (container_name == NULL) {
          +    return INVALID_DOCKER_CONTAINER_NAME;
          +  }
          
          We validate the container_id name in docker inspect, but not in docker rm, stop, or run
          

          Fixed.


          docker-util.c
          
          +static int add_param_to_command(const struct configuration *command_config, const char *key, const char *param,
          +                                 const int with_argument, char *out, const size_t outlen) {
          +  size_t tmp_buffer_size = 1024;
          
          What’s the reasoning behind the buffer size here? It’s hardcoded to 1024 in a bunch of places. This doesn’t follow the system path limit, the EXECUTOR_PATH_MAX or the system argument length.
          

          It's just a temporary buffer. The size doesn't matter too much because quote_and_append_arg will resize the buffer if it's too small.


          docker-util.c:set_network()
          
          +  } else if (value == NULL) {
          +    ret = 0;
          +    goto free_and_exit;
          
          Since we didn’t find any network in the configuration, we didn’t set a network. That seems like it should print out an error and/or fail. Right now it will just silently do nothing.
          

          You can launch docker containers with no network specified. It'll just pick up the default network.


          docker-util.c:set_network()
          
          +  if (permitted_values != NULL) {
          +    free(permitted_values[0]);
          +    free(permitted_values);
          +  }
          
          We need to loop through permitted_values and free everything or else we will leak.
          
          docker-util.c:normalize_mounts()
          
          +  char *alloc_ptr = mounts[0];
          +  for (i = 0; mounts[i] != NULL; ++i) {
          +    tmp = normalize_mount(mounts[i]);
          +    if (tmp == NULL) {
          +      return -1;
          +    }
          +    mounts[i] = tmp;
          +  }
          +  free(alloc_ptr);
          
          Why are we freeing mounts[0] or anything at all here?
          
          docker-util.c:add_mounts()
          
          +  if(permitted_ro_mounts != NULL) {
          +    free(permitted_ro_mounts[0]);
          +    free(permitted_ro_mounts);
          +  }
          +  if (permitted_rw_mounts != NULL) {
          +    for (i = 0; permitted_rw_mounts[i] != NULL; ++i) {
          +      free(permitted_rw_mounts[i]);
          +    }
          +    free(permitted_rw_mounts);
          +  }
          +  if (values != NULL) {
          +    free(values[0]);
          +    free(values);
          +  }
          
          You’ve done this above too, so I might be missing something. But I don’t see why we only want to free the 0th element of permitted_ro_mounts. Is there something that guarantees that permitted_ro_mounts will only have the first element of the pointer array populated? The pointer arrays are getting created by get_configuration_values_delimiter(), which ends up doing a strdup inside of a while loop via split_delimiter().
          
          docker-util.c:set_capabilities()
          
          +  if(values != NULL) {
          +    free(values[0]);
          +    free(values);
          +  }
          +  if(permitted_values != NULL) {
          +    free(permitted_values[0]);
          +    free(permitted_values);
          +  }
          
          Same issue with not looping when freeing
          
          docker-util.c:set_devices()
          
          +  if (permitted_devices != NULL) {
          +    free(permitted_devices[0]);
          +    free(permitted_devices);
          +  }
          +  if (values != NULL) {
          +    free(values[0]);
          +    free(values);
          
          Same issue with not looping when freeing
          
          docker-util.c:get_docker_run_command()
          
          +        if (ret != 0) {
          +          free(launch_command[0]);
          +          free(launch_command);
          +          free(tmp_buffer);
          
          Same issue with not looping when freeing for launch_command
          

          This was because I didn't realise the configuration behaviour had changed. Earlier it used to return the result of the strtok_r which got changed in a recent patch to the strdup. Thanks for catching it!


          docker-util.c:check_mount_permitted()
          
          +    // directory check
          +    permitted_mount_len = strlen(permitted_mounts[i]);
          +    if (permitted_mount_len > 0
          +        && permitted_mounts[i][permitted_mount_len - 1] == '/') {
          +      if (strncmp(requested, permitted_mounts[i], permitted_mount_len) == 0) {
          +        return 1;
          
          This looks potentially dangerous. Though we are normalizing the paths in all of the current invocations, someone in the future may add in another invocation that doesn’t normalize. I think at a bare minimum we need a comment saying that this function requires all symlinks to be resolved in the path that is passed to this function. Because if we allow symlinks to be followed then we will get different behavior because of how docker treats symlinks. A possible fix for this could be to combine get_src_mount() and get_real_src_mount() into a single function that parses the mount string to get the source and then resolves it.
          

          I've moved the normalize_mount call into the check_mount_permitted function. Does that address your concern?


          docker-util.c:get_real_src_mount()
          
          +  char *tmp = strdup(src_mount);
          +  if (tmp == NULL) {
          +    fprintf(ERRORFILE, "Could not allocate memory\n");
          +    exit(OUT_OF_MEMORY);
          +  }
          
          We should be consistent on our error handling. Some calls to strdup we exit with OOM and some we let it return error. This is inconsistent between get_src_mount() and get_real_src_mount(). I’m not sure I have a preference on whether to exit here or not, but we should make sure all places are consistent.
          On another note, is this function necessary given that we have normalize_mount()?
          

          Removed get_real_src_mount because as you pointed out, we call realpath in normalize_mount


          docker-util.c:add_mounts()
          
          +  if (real_src_mount != NULL) {
          +    free(real_src_mount);
          +  }
          +  if (src_mount != NULL) {
          +    free(src_mount);
          +  }
          +  if(normalized_src_mount != NULL) {
          +    free(normalized_src_mount);
          +  }
          +  if(container_executor_cfg_path != NULL) {
          +    free((char *) container_executor_cfg_path);
          +  }
          
          We don’t need to check for null. free will do nothing if a null pointer is passed, so it’s safe and makes the check unnecessary.
          

          Fixed.


          docker-util.c:add_mounts()
          
          +      free(real_src_mount);
          +      free(src_mount);
          +      free(normalized_src_mount);
          +      src_mount = NULL;
          +      real_src_mount = NULL;
          +      normalized_src_mount = NULL;
          
          Is the clear necessary here? The next time they are used in the loop they will be written instead of read. And the convention of the code in this file is to not clear after free (though maybe it should be?)
          

          This was so that the I can call free in the free_and_exit label without worrying about calling free on the same pointer.


          docker-util.c:add_mounts()
          
          +      real_src_mount = get_real_src_mount(src_mount);
          +      if (real_src_mount == NULL) {
          +        ret = INVALID_DOCKER_MOUNT;
          +        goto free_and_exit;
          +      }
          +      normalized_src_mount = normalize_mount(real_src_mount);
          +      if(normalized_src_mount == NULL) {
          +        ret = INVALID_DOCKER_MOUNT;
          +        goto  free_and_exit;
          +      }
          
          Do we need to call both get_real_src_mount() and normalize_mount()? As I mentioned in an earlier comment, get_real_src_mount() seems to be a subset of the functionality of normalize_mount(). They both call realpath(), but normalize_mount() does some extra directory stuff.
          

          You are correct. I removed the get_real_src_mount function.


          docker-util.c:set_privileged()
          
          +  if (value != NULL) {
          +    free(value);
          +  }
          +  if (privileged_container_enabled != NULL) {
          +    free(privileged_container_enabled);
          +  }
          
          Don’t need to do the null check before free
          

          Fixed.


          docker-util.c:set_capabilities()
          
          +  if (values != NULL) {
          +    if(permitted_values != NULL) {
          +      for (i = 0; values[i] != NULL; ++i) {
          +        memset(tmp_buffer, 0, tmp_buffer_size);
          +        permitted = 0;
          +        for (j = 0; permitted_values[j] != NULL; ++j) {
          +          if (strcmp(values[i], permitted_values[j]) == 0) {
          +            permitted = 1;
          +            break;
          +          }
          +        }
          
          I think this is the 3rd time I’ve seen this structure. I think it would be nice if we had a single function that took in a double array for permitted_values, and the value to be checked to see whether that value is permitted. That way we don’t have to duplicate all of the looping and can minimize the amount of code that we need to maintain. We could use this for networks, capabilities, mounts, devices, etc.
          

          Fixed. I added a new function called add_param_to_command_if_allowed which if used for network, capabilities and devices. Mounts has it's own function due to the read-write behaviour.


          docker-util.c:get_docker_run_command()
          
          +  memset(out, 0, outlen);
          
          Is there a reason we need to memset out here but not in the other get_*_command functions?
          

          Not needed, fixed.


          docker-util.c:get_docker_run_command()
          
          +    quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "", image);
          +    ret = add_to_buffer(out, outlen, tmp_buffer);
          +    if (ret != 0) {
          +      return BUFFER_TOO_SMALL;
          +    }
          +    memset(tmp_buffer, 0, tmp_buffer_size);
          +
          +    launch_command = get_configuration_values_delimiter("launch-command", DOCKER_COMMAND_FILE_SECTION, &command_config,
          +                                                        ",");
          +    if (launch_command != NULL) {
          +      for (i = 0; launch_command[i] != NULL; ++i) {
          +        memset(tmp_buffer, 0, tmp_buffer_size);
          
          Don’t think we need the first memset here since the first thing we do in the for loop is memset.
          

          Fixed.


          docker-util.c:get_docker_run_command()
          
          +  ret = add_to_buffer(out, outlen, DOCKER_RUN_COMMAND);
          +  if (ret == 0) {
          …
          +  }
          +  return BUFFER_TOO_SMALL;
          
          Rather than put this whole block in an if statement, we could invert the if statement and have an early return on failure, similar to what we do a few lines up with add_docker_config_param()
          

          Fixed.


          As a general comment for docker-util.c, it would be nice if we could consolidate some of the code in the get_command functions and in the set functions. There is a lot of redundant code in there. I’m worried that someone in the future will change part of the flow in one function (possibly a bug fix) that will then not get propagated to the rest of the similar code. This wouldn't necessarily have to be a part of this patch, but it might get kicked down the line and never get in if we don't do it now.

          I think I've addressed some of this as part of the latest patch. Can you take a look and let me know?

          Thank you the comprehensive review. Very much appreciated.

          Show
          vvasudev Varun Vasudev added a comment - Thank you for the review Eric Badger ! Should we fix the no newline at the end of file warnings? The apply tool complains about them. Fixed. DockerCommand:getCommandArguments() public Map<String, List<String>> getCommandArguments() { return Collections.unmodifiableMap(commandArguments); } This will return the command as well as the arguments. Unless we are considering the /usr/docker to be the actual command and inspect to be one of the arguments. If that’s what we’re expecting to happen here, then the name is a little bit misleading. This might be more of a problem with how commandArguments is named than how this function is named. Renamed function to getCommandWithArguments. Is that ok? container-executor.c:construct_docker_command() +char *construct_docker_command(const char *command_file) { + int ret = 0; + char *buffer = (char *) malloc(EXECUTOR_PATH_MAX * sizeof(char)); This should use _SC_ARG_MAX as we did in YARN-6988 size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024); Also, why not use calloc() instead of malloc() and then memset()? container-executor.c:run_docker() Fixed; changed to use the alloc_and_clear function. + docker_command = construct_docker_command(command_file); + docker_binary = get_docker_binary(&CFG); I don’t see these getting freed. Am I missing this invocation somewhere? container-executor.c:run_docker() We call the execvp function, the running program will get replaced by the docker invocation. + memset(docker_command_with_binary, 0, EXECUTOR_PATH_MAX); Is this necessary? We allocate the memory with calloc() which already clears all of the memory upon allocation. Yep. Fixed. { container-executor.h // get the executable's filename char* get_executable(char *argv0); Do we need this declaration (in container-executor.h) since we have moved that declaration into get_executable.h? We should also add get_executable.h in the appropriate places. Looks like main.c and test-container-executor.c both call get_executable. You're correct; fixed main.c:assert_valid_setup() - fprintf(ERRORFILE,"realpath of executable: %s\n", - errno != 0 ? strerror(errno) : "unknown"); - flush_and_close_log_files(); - exit(-1); + fprintf(ERRORFILE, "realpath of executable: %s\n", + errno != 0 ? strerror(errno) : "unknown"); + exit(INVALID_CONFIG_FILE); Why don’t we want to flush the log files anymore? Fixed. util.c:alloc_and_clear_memory() +void* alloc_and_clear_memory(size_t num, size_t size) { + void *ret = calloc(num, size); + if (ret == NULL) { + exit(OUT_OF_MEMORY); + } + return ret; +} Should we inline this? Also, if we’re going to use this function, then all calloc calls should be replaced with this (like the ones I mentioned above) util.h Fixed(made function inline and replaced calloc invocations with alloc_and_clear) // DOCKER_CONTAINER_NAME_INVALID = 41, Should add (NOT USED) to follow convention docker-util.c:read_and_verify_command_file() Fixed. if (command == NULL || (strcmp(command, docker_command) != 0)) { ret = INCORRECT_COMMAND; } Is command guaranteed to be null-terminated? It comes from the configuration file, which is a Java creation and I don’t think Java null-terminates. This comment is probably relevant for quite a few other places that are doing string operations. We need to be very safe about this or else we risk possibly overrunning into random regions of the heap. A safe alternative would be to use the “n” version of all the string operations. This patch uses a mixed bag of the regular versions and their accompanying “n” versions. I don’t quite understand the reasoning behind the usage of each. If we guarantee that the string is null terminated (and always null terminated) then we don’t need the “n” versions. But even if we guarantee that the input string is null terminated, it may accidentally have the null character chopped off by an off by 1 error in a strdup or something like that. So my preference here would be to use the “n” versions of all of the string functions. Thoughts? command is guaranteed to be null terminated by the confiugration functions. If we use the 'n' versions of the function we end up doing a 'begins with' match instead of an exact match which could cause problems(e.g. "inspect" would match "inspectcommand") docker-util.c:read_and_verify_command_file() + if (current_len + string_len < bufflen - 1) { + strncpy(buff + current_len, string, bufflen - current_len); + buff[current_len + string_len] = '\0'; + return 0; + } Here it is copying over bufflen - current_len bytes, when we really only have string_len bytes to copy. It’s not going to overflow buff, but we might copy some extra garbage past the end of string if it’s not null terminated. Good point, fixed. docker-util.c: add_docker_config_param() +static int add_docker_config_param(const struct configuration *command_config, char *out, const size_t outlen) { + int ret = 0; + size_t tmp_buffer_size = 4096; + char *tmp_buffer; + char *config = get_configuration_value("docker-config", DOCKER_COMMAND_FILE_SECTION, command_config); + if (config != NULL) { + tmp_buffer = (char *) alloc_and_clear_memory(tmp_buffer_size, sizeof(char)); + quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "--config=", config); + ret = add_to_buffer(out, outlen, tmp_buffer); + free(tmp_buffer); + free(config); + } + return ret; Is this function necessary? Can’t it be done in a call to add_param_to_command? You're correct again . Fixed. docker-util.c: get_docker_load_command() + image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config); + if (image_name == NULL) { + return INVALID_DOCKER_IMAGE_NAME; + } + + memset(out, 0, outlen); Maybe being paranoid is a good thing, but get_docker_load_command and any of the other commands from get_docker_command are always passed in a buffer that has already been nulled out, so the memset here is redundant. I imagine it will be similar in the other get_*_comand functions. Maybe we should put a comment saying that it is the caller’s responsibility to null out the buffer that they pass in and get rid of these memsets? Either that or keep the memset here and get rid of the resets higher up in get_docker_command? I've gotten rid of the memset in the higher calls and left the memset in the get_docker_* functions. I just feel more comfortable having the memset there. docker-util.c: get_docker_load_command() + image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config); + if (image_name == NULL) { + return INVALID_DOCKER_IMAGE_NAME; + } We should validate the image name here as we do in get_docker_pull_command below. You can pass tar files to docker load so we can't do much validation here. Docker save lets you save the image to any file you want and then you load it using docker load. docker-util.c: get_docker_rm_command() + container_name = get_configuration_value("name", DOCKER_COMMAND_FILE_SECTION, &command_config); + if (container_name == NULL) { + return INVALID_DOCKER_CONTAINER_NAME; + } We validate the container_id name in docker inspect, but not in docker rm, stop, or run Fixed. docker-util.c +static int add_param_to_command(const struct configuration *command_config, const char *key, const char *param, + const int with_argument, char *out, const size_t outlen) { + size_t tmp_buffer_size = 1024; What’s the reasoning behind the buffer size here? It’s hardcoded to 1024 in a bunch of places. This doesn’t follow the system path limit, the EXECUTOR_PATH_MAX or the system argument length. It's just a temporary buffer. The size doesn't matter too much because quote_and_append_arg will resize the buffer if it's too small. docker-util.c:set_network() + } else if (value == NULL) { + ret = 0; + goto free_and_exit; Since we didn’t find any network in the configuration, we didn’t set a network. That seems like it should print out an error and/or fail. Right now it will just silently do nothing. You can launch docker containers with no network specified. It'll just pick up the default network. docker-util.c:set_network() + if (permitted_values != NULL) { + free(permitted_values[0]); + free(permitted_values); + } We need to loop through permitted_values and free everything or else we will leak. docker-util.c:normalize_mounts() + char *alloc_ptr = mounts[0]; + for (i = 0; mounts[i] != NULL; ++i) { + tmp = normalize_mount(mounts[i]); + if (tmp == NULL) { + return -1; + } + mounts[i] = tmp; + } + free(alloc_ptr); Why are we freeing mounts[0] or anything at all here? docker-util.c:add_mounts() + if(permitted_ro_mounts != NULL) { + free(permitted_ro_mounts[0]); + free(permitted_ro_mounts); + } + if (permitted_rw_mounts != NULL) { + for (i = 0; permitted_rw_mounts[i] != NULL; ++i) { + free(permitted_rw_mounts[i]); + } + free(permitted_rw_mounts); + } + if (values != NULL) { + free(values[0]); + free(values); + } You’ve done this above too, so I might be missing something. But I don’t see why we only want to free the 0th element of permitted_ro_mounts. Is there something that guarantees that permitted_ro_mounts will only have the first element of the pointer array populated? The pointer arrays are getting created by get_configuration_values_delimiter(), which ends up doing a strdup inside of a while loop via split_delimiter(). docker-util.c:set_capabilities() + if(values != NULL) { + free(values[0]); + free(values); + } + if(permitted_values != NULL) { + free(permitted_values[0]); + free(permitted_values); + } Same issue with not looping when freeing docker-util.c:set_devices() + if (permitted_devices != NULL) { + free(permitted_devices[0]); + free(permitted_devices); + } + if (values != NULL) { + free(values[0]); + free(values); Same issue with not looping when freeing docker-util.c:get_docker_run_command() + if (ret != 0) { + free(launch_command[0]); + free(launch_command); + free(tmp_buffer); Same issue with not looping when freeing for launch_command This was because I didn't realise the configuration behaviour had changed. Earlier it used to return the result of the strtok_r which got changed in a recent patch to the strdup. Thanks for catching it! docker-util.c:check_mount_permitted() + // directory check + permitted_mount_len = strlen(permitted_mounts[i]); + if (permitted_mount_len > 0 + && permitted_mounts[i][permitted_mount_len - 1] == '/') { + if (strncmp(requested, permitted_mounts[i], permitted_mount_len) == 0) { + return 1; This looks potentially dangerous. Though we are normalizing the paths in all of the current invocations, someone in the future may add in another invocation that doesn’t normalize. I think at a bare minimum we need a comment saying that this function requires all symlinks to be resolved in the path that is passed to this function. Because if we allow symlinks to be followed then we will get different behavior because of how docker treats symlinks. A possible fix for this could be to combine get_src_mount() and get_real_src_mount() into a single function that parses the mount string to get the source and then resolves it. I've moved the normalize_mount call into the check_mount_permitted function. Does that address your concern? docker-util.c:get_real_src_mount() + char *tmp = strdup(src_mount); + if (tmp == NULL) { + fprintf(ERRORFILE, "Could not allocate memory\n"); + exit(OUT_OF_MEMORY); + } We should be consistent on our error handling. Some calls to strdup we exit with OOM and some we let it return error. This is inconsistent between get_src_mount() and get_real_src_mount(). I’m not sure I have a preference on whether to exit here or not, but we should make sure all places are consistent. On another note, is this function necessary given that we have normalize_mount()? Removed get_real_src_mount because as you pointed out, we call realpath in normalize_mount docker-util.c:add_mounts() + if (real_src_mount != NULL) { + free(real_src_mount); + } + if (src_mount != NULL) { + free(src_mount); + } + if(normalized_src_mount != NULL) { + free(normalized_src_mount); + } + if(container_executor_cfg_path != NULL) { + free((char *) container_executor_cfg_path); + } We don’t need to check for null. free will do nothing if a null pointer is passed, so it’s safe and makes the check unnecessary. Fixed. docker-util.c:add_mounts() + free(real_src_mount); + free(src_mount); + free(normalized_src_mount); + src_mount = NULL; + real_src_mount = NULL; + normalized_src_mount = NULL; Is the clear necessary here? The next time they are used in the loop they will be written instead of read. And the convention of the code in this file is to not clear after free (though maybe it should be?) This was so that the I can call free in the free_and_exit label without worrying about calling free on the same pointer. docker-util.c:add_mounts() + real_src_mount = get_real_src_mount(src_mount); + if (real_src_mount == NULL) { + ret = INVALID_DOCKER_MOUNT; + goto free_and_exit; + } + normalized_src_mount = normalize_mount(real_src_mount); + if(normalized_src_mount == NULL) { + ret = INVALID_DOCKER_MOUNT; + goto free_and_exit; + } Do we need to call both get_real_src_mount() and normalize_mount()? As I mentioned in an earlier comment, get_real_src_mount() seems to be a subset of the functionality of normalize_mount(). They both call realpath(), but normalize_mount() does some extra directory stuff. You are correct. I removed the get_real_src_mount function. docker-util.c:set_privileged() + if (value != NULL) { + free(value); + } + if (privileged_container_enabled != NULL) { + free(privileged_container_enabled); + } Don’t need to do the null check before free Fixed. docker-util.c:set_capabilities() + if (values != NULL) { + if(permitted_values != NULL) { + for (i = 0; values[i] != NULL; ++i) { + memset(tmp_buffer, 0, tmp_buffer_size); + permitted = 0; + for (j = 0; permitted_values[j] != NULL; ++j) { + if (strcmp(values[i], permitted_values[j]) == 0) { + permitted = 1; + break; + } + } I think this is the 3rd time I’ve seen this structure. I think it would be nice if we had a single function that took in a double array for permitted_values, and the value to be checked to see whether that value is permitted. That way we don’t have to duplicate all of the looping and can minimize the amount of code that we need to maintain. We could use this for networks, capabilities, mounts, devices, etc. Fixed. I added a new function called add_param_to_command_if_allowed which if used for network, capabilities and devices. Mounts has it's own function due to the read-write behaviour. docker-util.c:get_docker_run_command() + memset(out, 0, outlen); Is there a reason we need to memset out here but not in the other get_*_command functions? Not needed, fixed. docker-util.c:get_docker_run_command() + quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "", image); + ret = add_to_buffer(out, outlen, tmp_buffer); + if (ret != 0) { + return BUFFER_TOO_SMALL; + } + memset(tmp_buffer, 0, tmp_buffer_size); + + launch_command = get_configuration_values_delimiter("launch-command", DOCKER_COMMAND_FILE_SECTION, &command_config, + ","); + if (launch_command != NULL) { + for (i = 0; launch_command[i] != NULL; ++i) { + memset(tmp_buffer, 0, tmp_buffer_size); Don’t think we need the first memset here since the first thing we do in the for loop is memset. Fixed. docker-util.c:get_docker_run_command() + ret = add_to_buffer(out, outlen, DOCKER_RUN_COMMAND); + if (ret == 0) { … + } + return BUFFER_TOO_SMALL; Rather than put this whole block in an if statement, we could invert the if statement and have an early return on failure, similar to what we do a few lines up with add_docker_config_param() Fixed. As a general comment for docker-util.c, it would be nice if we could consolidate some of the code in the get_ command functions and in the set functions. There is a lot of redundant code in there. I’m worried that someone in the future will change part of the flow in one function (possibly a bug fix) that will then not get propagated to the rest of the similar code. This wouldn't necessarily have to be a part of this patch, but it might get kicked down the line and never get in if we don't do it now. I think I've addressed some of this as part of the latest patch. Can you take a look and let me know? Thank you the comprehensive review. Very much appreciated.
          Hide
          ebadger Eric Badger added a comment -

          Hey Varun Vasudev, I’ve finished my comments on the production code portion of the patch. This doesn’t include test code. I think I’ll wait for your response to this before I review the tests. I tried to be quite thorough and so I have quite a few comments. I hope that they are all correct interpretations, but it’s quite a big patch so I might have misunderstood some stuff and/or missed some stuff.


          Should we fix the no newline at the end of file warnings? The apply tool complains about them.


          DockerCommand:getCommandArguments()
          public Map<String, List<String>> getCommandArguments() {
            return Collections.unmodifiableMap(commandArguments);
          }
          

          This will return the command as well as the arguments. Unless we are considering the /usr/docker to be the actual command and inspect to be one of the arguments. If that’s what we’re expecting to happen here, then the name is a little bit misleading. This might be more of a problem with how commandArguments is named than how this function is named.


          container-executor.c:construct_docker_command()
          +char *construct_docker_command(const char *command_file) {
          +  int ret = 0;
          +  char *buffer = (char *) malloc(EXECUTOR_PATH_MAX * sizeof(char));
          

          This should use _SC_ARG_MAX as we did in YARN-6988
          size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024);
          Also, why not use calloc() instead of malloc() and then memset()?


          container-executor.c:run_docker()
          +  docker_command = construct_docker_command(command_file);
          +  docker_binary = get_docker_binary(&CFG);
          

          I don’t see these getting freed. Am I missing this invocation somewhere?


          container-executor.c:run_docker()
          +  memset(docker_command_with_binary, 0, EXECUTOR_PATH_MAX);
          

          Is this necessary? We allocate the memory with calloc() which already clears all of the memory upon allocation.


          container-executor.h
          // get the executable's filename
          char* get_executable(char *argv0);
          

          Do we need this declaration (in container-executor.h) since we have moved that declaration into get_executable.h? We should also add get_executable.h in the appropriate places. Looks like main.c and test-container-executor.c both call get_executable.


          main.c:assert_valid_setup()
          -    fprintf(ERRORFILE,"realpath of executable: %s\n",
          -      errno != 0 ? strerror(errno) : "unknown");
          -    flush_and_close_log_files();
          -    exit(-1);
          +    fprintf(ERRORFILE, "realpath of executable: %s\n",
          +            errno != 0 ? strerror(errno) : "unknown");
          +    exit(INVALID_CONFIG_FILE);
          

          Why don’t we want to flush the log files anymore?


          util.c:alloc_and_clear_memory()
          +void* alloc_and_clear_memory(size_t num, size_t size) {
          +  void *ret = calloc(num, size);
          +  if (ret == NULL) {
          +    exit(OUT_OF_MEMORY);
          +  }
          +  return ret;
          +}
          

          Should we inline this? Also, if we’re going to use this function, then all calloc calls should be replaced with this (like the ones I mentioned above)


          util.h
          // DOCKER_CONTAINER_NAME_INVALID = 41,
          

          Should add (NOT USED) to follow convention


          docker-util.c:read_and_verify_command_file()
          if (command == NULL || (strcmp(command, docker_command) != 0)) {
            ret = INCORRECT_COMMAND;
          }
          

          Is command guaranteed to be null-terminated? It comes from the configuration file, which is a Java creation and I don’t think Java null-terminates. This comment is probably relevant for quite a few other places that are doing string operations. We need to be very safe about this or else we risk possibly overrunning into random regions of the heap. A safe alternative would be to use the “n” version of all the string operations. This patch uses a mixed bag of the regular versions and their accompanying “n” versions. I don’t quite understand the reasoning behind the usage of each. If we guarantee that the string is null terminated (and always null terminated) then we don’t need the “n” versions. But even if we guarantee that the input string is null terminated, it may accidentally have the null character chopped off by an off by 1 error in a strdup or something like that. So my preference here would be to use the “n” versions of all of the string functions. Thoughts?


          docker-util.c:read_and_verify_command_file()
          +  if (current_len + string_len < bufflen - 1) {
          +    strncpy(buff + current_len, string, bufflen - current_len);
          +    buff[current_len + string_len] = '\0';
          +    return 0;
          +  }
          

          Here it is copying over bufflen - current_len bytes, when we really only have string_len bytes to copy. It’s not going to overflow buff, but we might copy some extra garbage past the end of string if it’s not null terminated.


          docker-util.c: add_docker_config_param()
          +static int add_docker_config_param(const struct configuration *command_config, char *out, const size_t outlen) {
          +  int ret = 0;
          +  size_t tmp_buffer_size = 4096;
          +  char *tmp_buffer;
          +  char *config = get_configuration_value("docker-config", DOCKER_COMMAND_FILE_SECTION, command_config);
          +  if (config != NULL) {
          +    tmp_buffer = (char *) alloc_and_clear_memory(tmp_buffer_size, sizeof(char));
          +    quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "--config=", config);
          +    ret = add_to_buffer(out, outlen, tmp_buffer);
          +    free(tmp_buffer);
          +    free(config);
          +  }
          +  return ret;
          

          Is this function necessary? Can’t it be done in a call to add_param_to_command?


          docker-util.c: get_docker_load_command()
          +  image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config);
          +  if (image_name == NULL) {
          +    return INVALID_DOCKER_IMAGE_NAME;
          +  }
          +
          +  memset(out, 0, outlen);
          

          Maybe being paranoid is a good thing, but get_docker_load_command and any of the other commands from get_docker_command are always passed in a buffer that has already been nulled out, so the memset here is redundant. I imagine it will be similar in the other get_*_comand functions. Maybe we should put a comment saying that it is the caller’s responsibility to null out the buffer that they pass in and get rid of these memsets? Either that or keep the memset here and get rid of the resets higher up in get_docker_command?


          docker-util.c: get_docker_load_command()
          +  image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config);
          +  if (image_name == NULL) {
          +    return INVALID_DOCKER_IMAGE_NAME;
          +  }
          

          We should validate the image name here as we do in get_docker_pull_command below.


          docker-util.c: get_docker_rm_command()
          +  container_name = get_configuration_value("name", DOCKER_COMMAND_FILE_SECTION, &command_config);
          +  if (container_name == NULL) {
          +    return INVALID_DOCKER_CONTAINER_NAME;
          +  }
          

          We validate the container_id name in docker inspect, but not in docker rm, stop, or run


          docker-util.c
          +static int add_param_to_command(const struct configuration *command_config, const char *key, const char *param,
          +                                 const int with_argument, char *out, const size_t outlen) {
          +  size_t tmp_buffer_size = 1024;
          

          What’s the reasoning behind the buffer size here? It’s hardcoded to 1024 in a bunch of places. This doesn’t follow the system path limit, the EXECUTOR_PATH_MAX or the system argument length.


          docker-util.c:set_network()
          +  } else if (value == NULL) {
          +    ret = 0;
          +    goto free_and_exit;
          

          Since we didn’t find any network in the configuration, we didn’t set a network. That seems like it should print out an error and/or fail. Right now it will just silently do nothing.


          docker-util.c:set_network()
          +  if (permitted_values != NULL) {
          +    free(permitted_values[0]);
          +    free(permitted_values);
          +  }
          

          We need to loop through permitted_values and free everything or else we will leak.


          docker-util.c:check_mount_permitted()
          +    // directory check
          +    permitted_mount_len = strlen(permitted_mounts[i]);
          +    if (permitted_mount_len > 0
          +        && permitted_mounts[i][permitted_mount_len - 1] == '/') {
          +      if (strncmp(requested, permitted_mounts[i], permitted_mount_len) == 0) {
          +        return 1;
          

          This looks potentially dangerous. Though we are normalizing the paths in all of the current invocations, someone in the future may add in another invocation that doesn’t normalize. I think at a bare minimum we need a comment saying that this function requires all symlinks to be resolved in the path that is passed to this function. Because if we allow symlinks to be followed then we will get different behavior because of how docker treats symlinks. A possible fix for this could be to combine get_src_mount() and get_real_src_mount() into a single function that parses the mount string to get the source and then resolves it.


          docker-util.c:normalize_mounts()
          +  char *alloc_ptr = mounts[0];
          +  for (i = 0; mounts[i] != NULL; ++i) {
          +    tmp = normalize_mount(mounts[i]);
          +    if (tmp == NULL) {
          +      return -1;
          +    }
          +    mounts[i] = tmp;
          +  }
          +  free(alloc_ptr);
          

          Why are we freeing mounts[0] or anything at all here?


          docker-util.c:get_real_src_mount()
          +  char *tmp = strdup(src_mount);
          +  if (tmp == NULL) {
          +    fprintf(ERRORFILE, "Could not allocate memory\n");
          +    exit(OUT_OF_MEMORY);
          +  }
          

          We should be consistent on our error handling. Some calls to strdup we exit with OOM and some we let it return error. This is inconsistent between get_src_mount() and get_real_src_mount(). I’m not sure I have a preference on whether to exit here or not, but we should make sure all places are consistent.
          On another note, is this function necessary given that we have normalize_mount()?


          docker-util.c:add_mounts()
          +  if(permitted_ro_mounts != NULL) {
          +    free(permitted_ro_mounts[0]);
          +    free(permitted_ro_mounts);
          +  }
          +  if (permitted_rw_mounts != NULL) {
          +    for (i = 0; permitted_rw_mounts[i] != NULL; ++i) {
          +      free(permitted_rw_mounts[i]);
          +    }
          +    free(permitted_rw_mounts);
          +  }
          +  if (values != NULL) {
          +    free(values[0]);
          +    free(values);
          +  }
          

          You’ve done this above too, so I might be missing something. But I don’t see why we only want to free the 0th element of permitted_ro_mounts. Is there something that guarantees that permitted_ro_mounts will only have the first element of the pointer array populated? The pointer arrays are getting created by get_configuration_values_delimiter(), which ends up doing a strdup inside of a while loop via split_delimiter().


          docker-util.c:add_mounts()
          +  if (real_src_mount != NULL) {
          +    free(real_src_mount);
          +  }
          +  if (src_mount != NULL) {
          +    free(src_mount);
          +  }
          +  if(normalized_src_mount != NULL) {
          +    free(normalized_src_mount);
          +  }
          +  if(container_executor_cfg_path != NULL) {
          +    free((char *) container_executor_cfg_path);
          +  }
          

          We don’t need to check for null. free will do nothing if a null pointer is passed, so it’s safe and makes the check unnecessary.


          docker-util.c:add_mounts()
          +      free(real_src_mount);
          +      free(src_mount);
          +      free(normalized_src_mount);
          +      src_mount = NULL;
          +      real_src_mount = NULL;
          +      normalized_src_mount = NULL;
          

          Is the clear necessary here? The next time they are used in the loop they will be written instead of read. And the convention of the code in this file is to not clear after free (though maybe it should be?)


          docker-util.c:add_mounts()
          +      real_src_mount = get_real_src_mount(src_mount);
          +      if (real_src_mount == NULL) {
          +        ret = INVALID_DOCKER_MOUNT;
          +        goto free_and_exit;
          +      }
          +      normalized_src_mount = normalize_mount(real_src_mount);
          +      if(normalized_src_mount == NULL) {
          +        ret = INVALID_DOCKER_MOUNT;
          +        goto  free_and_exit;
          +      }
          

          Do we need to call both get_real_src_mount() and normalize_mount()? As I mentioned in an earlier comment, get_real_src_mount() seems to be a subset of the functionality of normalize_mount(). They both call realpath(), but normalize_mount() does some extra directory stuff.


          docker-util.c:add_mounts()
          +          // determine if the user can modify the container-executor.cfg file
          +          tmp_path_buffer[0] = normalized_src_mount;
          +          // just re-use the function, flip the args to check if the container-executor path is in the requested
          +          // mount point
          +          ret = check_mount_permitted(tmp_path_buffer, container_executor_cfg_path);
          +          if (ret == 1) {
          +            fprintf(ERRORFILE, "Attempting to mount a parent directory '%s' of container-executor.cfg as read-write\n",
          +                    values[i]);
          +            ret = INVALID_DOCKER_RW_MOUNT;
          +            goto free_and_exit;
          +          }
          

          I understand the idea here, but isn’t this at the discretion of the admin? What if clusters have container-executor.cfg in a conf directory with a bunch of other configs? This will cause them to have to make major changes in the structure of their configs and deployments. Especially if they don’t have security enabled, they might not care about these potential vulnerabilities. Plus, unix permissions have the container-executor.cfg file as only readable by root. This would be a change that the administrator couldn’t override to relax the constraint. So while I like the idea of trying to protect container-executor.cfg, I think this should be something that can be relaxed by the admins at their discretion.


          docker-util.c:set_privileged()
          +  if (value != NULL) {
          +    free(value);
          +  }
          +  if (privileged_container_enabled != NULL) {
          +    free(privileged_container_enabled);
          +  }
          

          Don’t need to do the null check before free


          docker-util.c:set_capabilities()
          +  if (values != NULL) {
          +    if(permitted_values != NULL) {
          +      for (i = 0; values[i] != NULL; ++i) {
          +        memset(tmp_buffer, 0, tmp_buffer_size);
          +        permitted = 0;
          +        for (j = 0; permitted_values[j] != NULL; ++j) {
          +          if (strcmp(values[i], permitted_values[j]) == 0) {
          +            permitted = 1;
          +            break;
          +          }
          +        }
          

          I think this is the 3rd time I’ve seen this structure. I think it would be nice if we had a single function that took in a double array for permitted_values, and the value to be checked to see whether that value is permitted. That way we don’t have to duplicate all of the looping and can minimize the amount of code that we need to maintain. We could use this for networks, capabilities, mounts, devices, etc.


          docker-util.c:set_capabilities()
          +  if(values != NULL) {
          +    free(values[0]);
          +    free(values);
          +  }
          +  if(permitted_values != NULL) {
          +    free(permitted_values[0]);
          +    free(permitted_values);
          +  }
          

          Same issue with not looping when freeing


          docker-util.c:set_devices()
          +  if (permitted_devices != NULL) {
          +    free(permitted_devices[0]);
          +    free(permitted_devices);
          +  }
          +  if (values != NULL) {
          +    free(values[0]);
          +    free(values);
          

          Same issue with not looping when freeing


          docker-util.c:get_docker_run_command()
          +  memset(out, 0, outlen);
          

          Is there a reason we need to memset out here but not in the other get_*_command functions?


          docker-util.c:get_docker_run_command()
          +    quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "", image);
          +    ret = add_to_buffer(out, outlen, tmp_buffer);
          +    if (ret != 0) {
          +      return BUFFER_TOO_SMALL;
          +    }
          +    memset(tmp_buffer, 0, tmp_buffer_size);
          +
          +    launch_command = get_configuration_values_delimiter("launch-command", DOCKER_COMMAND_FILE_SECTION, &command_config,
          +                                                        ",");
          +    if (launch_command != NULL) {
          +      for (i = 0; launch_command[i] != NULL; ++i) {
          +        memset(tmp_buffer, 0, tmp_buffer_size);
          

          Don’t think we need the first memset here since the first thing we do in the for loop is memset.


          docker-util.c:get_docker_run_command()
          +  ret = add_to_buffer(out, outlen, DOCKER_RUN_COMMAND);
          +  if (ret == 0) {
          …
          +  }
          +  return BUFFER_TOO_SMALL;
          

          Rather than put this whole block in an if statement, we could invert the if statement and have an early return on failure, similar to what we do a few lines up with add_docker_config_param()


          docker-util.c:get_docker_run_command()
          +        if (ret != 0) {
          +          free(launch_command[0]);
          +          free(launch_command);
          +          free(tmp_buffer);
          

          Same issue with not looping when freeing for launch_command


          As a general comment for docker-util.c, it would be nice if we could consolidate some of the code in the get_*_command functions and in the set_* functions. There is a lot of redundant code in there. I’m worried that someone in the future will change part of the flow in one function (possibly a bug fix) that will then not get propagated to the rest of the similar code. This wouldn't necessarily have to be a part of this patch, but it might get kicked down the line and never get in if we don't do it now.

          Show
          ebadger Eric Badger added a comment - Hey Varun Vasudev , I’ve finished my comments on the production code portion of the patch. This doesn’t include test code. I think I’ll wait for your response to this before I review the tests. I tried to be quite thorough and so I have quite a few comments. I hope that they are all correct interpretations, but it’s quite a big patch so I might have misunderstood some stuff and/or missed some stuff. Should we fix the no newline at the end of file warnings? The apply tool complains about them. DockerCommand:getCommandArguments() public Map<String, List<String>> getCommandArguments() { return Collections.unmodifiableMap(commandArguments); } This will return the command as well as the arguments. Unless we are considering the /usr/docker to be the actual command and inspect to be one of the arguments. If that’s what we’re expecting to happen here, then the name is a little bit misleading. This might be more of a problem with how commandArguments is named than how this function is named. container-executor.c:construct_docker_command() +char *construct_docker_command(const char *command_file) { + int ret = 0; + char *buffer = (char *) malloc(EXECUTOR_PATH_MAX * sizeof(char)); This should use _SC_ARG_MAX as we did in YARN-6988 size_t command_size = MIN(sysconf(_SC_ARG_MAX), 128*1024); Also, why not use calloc() instead of malloc() and then memset() ? container-executor.c:run_docker() + docker_command = construct_docker_command(command_file); + docker_binary = get_docker_binary(&CFG); I don’t see these getting freed. Am I missing this invocation somewhere? container-executor.c:run_docker() + memset(docker_command_with_binary, 0, EXECUTOR_PATH_MAX); Is this necessary? We allocate the memory with calloc() which already clears all of the memory upon allocation. container-executor.h // get the executable's filename char* get_executable(char *argv0); Do we need this declaration (in container-executor.h) since we have moved that declaration into get_executable.h? We should also add get_executable.h in the appropriate places. Looks like main.c and test-container-executor.c both call get_executable . main.c:assert_valid_setup() - fprintf(ERRORFILE,"realpath of executable: %s\n", - errno != 0 ? strerror(errno) : "unknown"); - flush_and_close_log_files(); - exit(-1); + fprintf(ERRORFILE, "realpath of executable: %s\n", + errno != 0 ? strerror(errno) : "unknown"); + exit(INVALID_CONFIG_FILE); Why don’t we want to flush the log files anymore? util.c:alloc_and_clear_memory() +void* alloc_and_clear_memory(size_t num, size_t size) { + void *ret = calloc(num, size); + if (ret == NULL) { + exit(OUT_OF_MEMORY); + } + return ret; +} Should we inline this? Also, if we’re going to use this function, then all calloc calls should be replaced with this (like the ones I mentioned above) util.h // DOCKER_CONTAINER_NAME_INVALID = 41, Should add (NOT USED) to follow convention docker-util.c:read_and_verify_command_file() if (command == NULL || (strcmp(command, docker_command) != 0)) { ret = INCORRECT_COMMAND; } Is command guaranteed to be null-terminated? It comes from the configuration file, which is a Java creation and I don’t think Java null-terminates. This comment is probably relevant for quite a few other places that are doing string operations. We need to be very safe about this or else we risk possibly overrunning into random regions of the heap. A safe alternative would be to use the “n” version of all the string operations. This patch uses a mixed bag of the regular versions and their accompanying “n” versions. I don’t quite understand the reasoning behind the usage of each. If we guarantee that the string is null terminated (and always null terminated) then we don’t need the “n” versions. But even if we guarantee that the input string is null terminated, it may accidentally have the null character chopped off by an off by 1 error in a strdup or something like that. So my preference here would be to use the “n” versions of all of the string functions. Thoughts? docker-util.c:read_and_verify_command_file() + if (current_len + string_len < bufflen - 1) { + strncpy(buff + current_len, string, bufflen - current_len); + buff[current_len + string_len] = '\0'; + return 0; + } Here it is copying over bufflen - current_len bytes, when we really only have string_len bytes to copy. It’s not going to overflow buff , but we might copy some extra garbage past the end of string if it’s not null terminated. docker-util.c: add_docker_config_param() +static int add_docker_config_param(const struct configuration *command_config, char *out, const size_t outlen) { + int ret = 0; + size_t tmp_buffer_size = 4096; + char *tmp_buffer; + char *config = get_configuration_value("docker-config", DOCKER_COMMAND_FILE_SECTION, command_config); + if (config != NULL) { + tmp_buffer = (char *) alloc_and_clear_memory(tmp_buffer_size, sizeof(char)); + quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "--config=", config); + ret = add_to_buffer(out, outlen, tmp_buffer); + free(tmp_buffer); + free(config); + } + return ret; Is this function necessary? Can’t it be done in a call to add_param_to_command ? docker-util.c: get_docker_load_command() + image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config); + if (image_name == NULL) { + return INVALID_DOCKER_IMAGE_NAME; + } + + memset(out, 0, outlen); Maybe being paranoid is a good thing, but get_docker_load_command and any of the other commands from get_docker_command are always passed in a buffer that has already been nulled out, so the memset here is redundant. I imagine it will be similar in the other get_*_comand functions. Maybe we should put a comment saying that it is the caller’s responsibility to null out the buffer that they pass in and get rid of these memsets? Either that or keep the memset here and get rid of the resets higher up in get_docker_command ? docker-util.c: get_docker_load_command() + image_name = get_configuration_value("image", DOCKER_COMMAND_FILE_SECTION, &command_config); + if (image_name == NULL) { + return INVALID_DOCKER_IMAGE_NAME; + } We should validate the image name here as we do in get_docker_pull_command below. docker-util.c: get_docker_rm_command() + container_name = get_configuration_value("name", DOCKER_COMMAND_FILE_SECTION, &command_config); + if (container_name == NULL) { + return INVALID_DOCKER_CONTAINER_NAME; + } We validate the container_id name in docker inspect, but not in docker rm, stop, or run docker-util.c +static int add_param_to_command(const struct configuration *command_config, const char *key, const char *param, + const int with_argument, char *out, const size_t outlen) { + size_t tmp_buffer_size = 1024; What’s the reasoning behind the buffer size here? It’s hardcoded to 1024 in a bunch of places. This doesn’t follow the system path limit, the EXECUTOR_PATH_MAX or the system argument length. docker-util.c:set_network() + } else if (value == NULL) { + ret = 0; + goto free_and_exit; Since we didn’t find any network in the configuration, we didn’t set a network. That seems like it should print out an error and/or fail. Right now it will just silently do nothing. docker-util.c:set_network() + if (permitted_values != NULL) { + free(permitted_values[0]); + free(permitted_values); + } We need to loop through permitted_values and free everything or else we will leak. docker-util.c:check_mount_permitted() + // directory check + permitted_mount_len = strlen(permitted_mounts[i]); + if (permitted_mount_len > 0 + && permitted_mounts[i][permitted_mount_len - 1] == '/') { + if (strncmp(requested, permitted_mounts[i], permitted_mount_len) == 0) { + return 1; This looks potentially dangerous. Though we are normalizing the paths in all of the current invocations, someone in the future may add in another invocation that doesn’t normalize. I think at a bare minimum we need a comment saying that this function requires all symlinks to be resolved in the path that is passed to this function. Because if we allow symlinks to be followed then we will get different behavior because of how docker treats symlinks. A possible fix for this could be to combine get_src_mount() and get_real_src_mount() into a single function that parses the mount string to get the source and then resolves it. docker-util.c:normalize_mounts() + char *alloc_ptr = mounts[0]; + for (i = 0; mounts[i] != NULL; ++i) { + tmp = normalize_mount(mounts[i]); + if (tmp == NULL) { + return -1; + } + mounts[i] = tmp; + } + free(alloc_ptr); Why are we freeing mounts[0] or anything at all here? docker-util.c:get_real_src_mount() + char *tmp = strdup(src_mount); + if (tmp == NULL) { + fprintf(ERRORFILE, "Could not allocate memory\n"); + exit(OUT_OF_MEMORY); + } We should be consistent on our error handling. Some calls to strdup we exit with OOM and some we let it return error. This is inconsistent between get_src_mount() and get_real_src_mount() . I’m not sure I have a preference on whether to exit here or not, but we should make sure all places are consistent. On another note, is this function necessary given that we have normalize_mount() ? docker-util.c:add_mounts() + if(permitted_ro_mounts != NULL) { + free(permitted_ro_mounts[0]); + free(permitted_ro_mounts); + } + if (permitted_rw_mounts != NULL) { + for (i = 0; permitted_rw_mounts[i] != NULL; ++i) { + free(permitted_rw_mounts[i]); + } + free(permitted_rw_mounts); + } + if (values != NULL) { + free(values[0]); + free(values); + } You’ve done this above too, so I might be missing something. But I don’t see why we only want to free the 0th element of permitted_ro_mounts . Is there something that guarantees that permitted_ro_mounts will only have the first element of the pointer array populated? The pointer arrays are getting created by get_configuration_values_delimiter() , which ends up doing a strdup inside of a while loop via split_delimiter() . docker-util.c:add_mounts() + if (real_src_mount != NULL) { + free(real_src_mount); + } + if (src_mount != NULL) { + free(src_mount); + } + if(normalized_src_mount != NULL) { + free(normalized_src_mount); + } + if(container_executor_cfg_path != NULL) { + free((char *) container_executor_cfg_path); + } We don’t need to check for null. free will do nothing if a null pointer is passed, so it’s safe and makes the check unnecessary. docker-util.c:add_mounts() + free(real_src_mount); + free(src_mount); + free(normalized_src_mount); + src_mount = NULL; + real_src_mount = NULL; + normalized_src_mount = NULL; Is the clear necessary here? The next time they are used in the loop they will be written instead of read. And the convention of the code in this file is to not clear after free (though maybe it should be?) docker-util.c:add_mounts() + real_src_mount = get_real_src_mount(src_mount); + if (real_src_mount == NULL) { + ret = INVALID_DOCKER_MOUNT; + goto free_and_exit; + } + normalized_src_mount = normalize_mount(real_src_mount); + if(normalized_src_mount == NULL) { + ret = INVALID_DOCKER_MOUNT; + goto free_and_exit; + } Do we need to call both get_real_src_mount() and normalize_mount() ? As I mentioned in an earlier comment, get_real_src_mount() seems to be a subset of the functionality of normalize_mount() . They both call realpath() , but normalize_mount() does some extra directory stuff. docker-util.c:add_mounts() + // determine if the user can modify the container-executor.cfg file + tmp_path_buffer[0] = normalized_src_mount; + // just re-use the function, flip the args to check if the container-executor path is in the requested + // mount point + ret = check_mount_permitted(tmp_path_buffer, container_executor_cfg_path); + if (ret == 1) { + fprintf(ERRORFILE, "Attempting to mount a parent directory '%s' of container-executor.cfg as read-write\n", + values[i]); + ret = INVALID_DOCKER_RW_MOUNT; + goto free_and_exit; + } I understand the idea here, but isn’t this at the discretion of the admin? What if clusters have container-executor.cfg in a conf directory with a bunch of other configs? This will cause them to have to make major changes in the structure of their configs and deployments. Especially if they don’t have security enabled, they might not care about these potential vulnerabilities. Plus, unix permissions have the container-executor.cfg file as only readable by root. This would be a change that the administrator couldn’t override to relax the constraint. So while I like the idea of trying to protect container-executor.cfg, I think this should be something that can be relaxed by the admins at their discretion. docker-util.c:set_privileged() + if (value != NULL) { + free(value); + } + if (privileged_container_enabled != NULL) { + free(privileged_container_enabled); + } Don’t need to do the null check before free docker-util.c:set_capabilities() + if (values != NULL) { + if(permitted_values != NULL) { + for (i = 0; values[i] != NULL; ++i) { + memset(tmp_buffer, 0, tmp_buffer_size); + permitted = 0; + for (j = 0; permitted_values[j] != NULL; ++j) { + if (strcmp(values[i], permitted_values[j]) == 0) { + permitted = 1; + break; + } + } I think this is the 3rd time I’ve seen this structure. I think it would be nice if we had a single function that took in a double array for permitted_values, and the value to be checked to see whether that value is permitted. That way we don’t have to duplicate all of the looping and can minimize the amount of code that we need to maintain. We could use this for networks, capabilities, mounts, devices, etc. docker-util.c:set_capabilities() + if(values != NULL) { + free(values[0]); + free(values); + } + if(permitted_values != NULL) { + free(permitted_values[0]); + free(permitted_values); + } Same issue with not looping when freeing docker-util.c:set_devices() + if (permitted_devices != NULL) { + free(permitted_devices[0]); + free(permitted_devices); + } + if (values != NULL) { + free(values[0]); + free(values); Same issue with not looping when freeing docker-util.c:get_docker_run_command() + memset(out, 0, outlen); Is there a reason we need to memset out here but not in the other get_*_command functions? docker-util.c:get_docker_run_command() + quote_and_append_arg(&tmp_buffer, &tmp_buffer_size, "", image); + ret = add_to_buffer(out, outlen, tmp_buffer); + if (ret != 0) { + return BUFFER_TOO_SMALL; + } + memset(tmp_buffer, 0, tmp_buffer_size); + + launch_command = get_configuration_values_delimiter("launch-command", DOCKER_COMMAND_FILE_SECTION, &command_config, + ","); + if (launch_command != NULL) { + for (i = 0; launch_command[i] != NULL; ++i) { + memset(tmp_buffer, 0, tmp_buffer_size); Don’t think we need the first memset here since the first thing we do in the for loop is memset. docker-util.c:get_docker_run_command() + ret = add_to_buffer(out, outlen, DOCKER_RUN_COMMAND); + if (ret == 0) { … + } + return BUFFER_TOO_SMALL; Rather than put this whole block in an if statement, we could invert the if statement and have an early return on failure, similar to what we do a few lines up with add_docker_config_param() docker-util.c:get_docker_run_command() + if (ret != 0) { + free(launch_command[0]); + free(launch_command); + free(tmp_buffer); Same issue with not looping when freeing for launch_command As a general comment for docker-util.c, it would be nice if we could consolidate some of the code in the get_*_command functions and in the set_* functions. There is a lot of redundant code in there. I’m worried that someone in the future will change part of the flow in one function (possibly a bug fix) that will then not get propagated to the rest of the similar code. This wouldn't necessarily have to be a part of this patch, but it might get kicked down the line and never get in if we don't do it now.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 18s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 46s Maven dependency ordering for branch
          +1 mvninstall 14m 20s trunk passed
          +1 compile 9m 1s trunk passed
          +1 checkstyle 0m 55s trunk passed
          +1 mvnsite 4m 7s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 23s trunk passed
                Patch Compile Tests
          0 mvndep 0m 12s Maven dependency ordering for patch
          +1 mvninstall 4m 5s the patch passed
          -1 compile 1m 44s hadoop-yarn in the patch failed.
          -1 cc 1m 44s hadoop-yarn in the patch failed.
          -1 javac 1m 44s hadoop-yarn in the patch failed.
          -0 checkstyle 0m 49s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 24 unchanged - 3 fixed = 25 total (was 27)
          +1 mvnsite 3m 38s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          +1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          +1 javadoc 2m 0s the patch passed
                Other Tests
          -1 unit 62m 17s hadoop-yarn in the patch failed.
          -1 unit 0m 53s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 36s The patch does not generate ASF License warnings.
          116m 57s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882883/YARN-6623.005.patch
          Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle
          uname Linux b86feff98082 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 267e19a
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          compile https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          cc https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17033/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17033/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 18s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 46s Maven dependency ordering for branch +1 mvninstall 14m 20s trunk passed +1 compile 9m 1s trunk passed +1 checkstyle 0m 55s trunk passed +1 mvnsite 4m 7s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 23s trunk passed       Patch Compile Tests 0 mvndep 0m 12s Maven dependency ordering for patch +1 mvninstall 4m 5s the patch passed -1 compile 1m 44s hadoop-yarn in the patch failed. -1 cc 1m 44s hadoop-yarn in the patch failed. -1 javac 1m 44s hadoop-yarn in the patch failed. -0 checkstyle 0m 49s hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 24 unchanged - 3 fixed = 25 total (was 27) +1 mvnsite 3m 38s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 51s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) +1 javadoc 2m 0s the patch passed       Other Tests -1 unit 62m 17s hadoop-yarn in the patch failed. -1 unit 0m 53s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 36s The patch does not generate ASF License warnings. 116m 57s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882883/YARN-6623.005.patch Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle uname Linux b86feff98082 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 267e19a Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html compile https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt cc https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-compile-hadoop-yarn-project_hadoop-yarn.txt checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/17033/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/17033/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/17033/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          Varun Vasudev, I started reviewing the .003 patch, but I see that you've updated a few times since then. So I'll post my comments so far and then continue my review with the most recent patch.

          Ran test-container-executor on both my mac and on rhel6 and it passed.

          1. Should feature.docker.enabled be moved to be with the rest of the docker configs?

          2. For the Docker configs, I think it'd be more clear if they were prefixed with docker.

          3. Since we're going to remove the hard-coded string in YARN-6968, should we leave the findbugs warning and take care of it there?

          4.

          +  /**
          +   * Add command commandWithArguments - this method is only meant for use by
          +   * sub-classes.
          +   *
          

          This is no longer just adding a command. Looks like it adds arguments to the command and adds the command as well if it doesn't exist. So the comment should be updated.

          5.

          +    this.commandArguments = new TreeMap<>();
          +    commandArguments.put(dockerCommandKey, new ArrayList<>());
          +    commandArguments.get(dockerCommandKey).add(command);
          

          I’m not sure I understand how this piece of code is supposed to work. In this method we add a new list with the key “docker-command”. But then in addCommandArguments() we add arguments via the key, which is always called in the subclasses with the name of the command being run (e.g. inspect). So the map for each DockerCommand will usually have 2 keys, “docker-command” and the command that actually needs arguments “inspect”. Is this how it’s intended to work or did I miss something here?

          6.

          +  protected final void addCommandArguments(String key, String value) {
          +    if (commandArguments.containsKey(key)) {
          +      commandArguments.get(key).add(value);
          +      return;
          +    }
          

          No need to call contains() and then call get(). We can just call get and then add the value if the return value isn't null.

          7.

          +  @Override
          +  public String toString() {
          +    StringBuffer ret = new StringBuffer("");
          +    ret.append(this.command);
          

          Any reason not to create the string with the command in the constructor? Something like
          StringBuffer ret = new StringBuffer(this.command);

          Show
          ebadger Eric Badger added a comment - Varun Vasudev , I started reviewing the .003 patch, but I see that you've updated a few times since then. So I'll post my comments so far and then continue my review with the most recent patch. Ran test-container-executor on both my mac and on rhel6 and it passed. 1. Should feature.docker.enabled be moved to be with the rest of the docker configs? 2. For the Docker configs, I think it'd be more clear if they were prefixed with docker. 3. Since we're going to remove the hard-coded string in YARN-6968 , should we leave the findbugs warning and take care of it there? 4. + /** + * Add command commandWithArguments - this method is only meant for use by + * sub-classes. + * This is no longer just adding a command. Looks like it adds arguments to the command and adds the command as well if it doesn't exist. So the comment should be updated. 5. + this.commandArguments = new TreeMap<>(); + commandArguments.put(dockerCommandKey, new ArrayList<>()); + commandArguments.get(dockerCommandKey).add(command); I’m not sure I understand how this piece of code is supposed to work. In this method we add a new list with the key “docker-command”. But then in addCommandArguments() we add arguments via the key , which is always called in the subclasses with the name of the command being run (e.g. inspect). So the map for each DockerCommand will usually have 2 keys, “docker-command” and the command that actually needs arguments “inspect”. Is this how it’s intended to work or did I miss something here? 6. + protected final void addCommandArguments(String key, String value) { + if (commandArguments.containsKey(key)) { + commandArguments.get(key).add(value); + return; + } No need to call contains() and then call get() . We can just call get and then add the value if the return value isn't null. 7. + @Override + public String toString() { + StringBuffer ret = new StringBuffer(""); + ret.append(this.command); Any reason not to create the string with the command in the constructor? Something like StringBuffer ret = new StringBuffer(this.command);
          Hide
          vvasudev Varun Vasudev added a comment -

          #define EXECUTOR_PATH_MAX 131072

          This is lots of allocation. The OS actually needs to zero all the allocated memory before giving it to other processes after close, so this might add to the overall CPU usage and memory bandwidth.

          Given that yarn local dirs are bind mounted into containers, using 4096 has proven problematic with even a small number of yarn local dirs/disks (3 in the case I saw). We're going to need to make the size of this buffer configurable. If it seems more appropriate to do so in a separate issue, I'm fine with that.

          This got fixed by YARN-6988.

          Show
          vvasudev Varun Vasudev added a comment - #define EXECUTOR_PATH_MAX 131072 This is lots of allocation. The OS actually needs to zero all the allocated memory before giving it to other processes after close, so this might add to the overall CPU usage and memory bandwidth. Given that yarn local dirs are bind mounted into containers, using 4096 has proven problematic with even a small number of yarn local dirs/disks (3 in the case I saw). We're going to need to make the size of this buffer configurable. If it seems more appropriate to do so in a separate issue, I'm fine with that. This got fixed by YARN-6988 .
          Hide
          vvasudev Varun Vasudev added a comment -

          New patch uploaded after rebasing to trunk.

          Show
          vvasudev Varun Vasudev added a comment - New patch uploaded after rebasing to trunk.
          Hide
          shanekumpf@gmail.com Shane Kumpf added a comment -

          #define EXECUTOR_PATH_MAX 131072

          This is lots of allocation. The OS actually needs to zero all the allocated memory before giving it to other processes after close, so this might add to the overall CPU usage and memory bandwidth.

          Given that yarn local dirs are bind mounted into containers, using 4096 has proven problematic with even a small number of yarn local dirs/disks (3 in the case I saw). We're going to need to make the size of this buffer configurable. If it seems more appropriate to do so in a separate issue, I'm fine with that.

          Image can be specified by a digest, which is more secure. I do not see that supported by the regex. IMAGE[:TAG|@DIGEST]

          This is the same regex used on the java side, just duplicated in c-e to address the concerns with direct invocation. We should fix this both places to support the digest notation. Again, this seems like a separate issue from what this patch is addressing.

          Show
          shanekumpf@gmail.com Shane Kumpf added a comment - #define EXECUTOR_PATH_MAX 131072 This is lots of allocation. The OS actually needs to zero all the allocated memory before giving it to other processes after close, so this might add to the overall CPU usage and memory bandwidth. Given that yarn local dirs are bind mounted into containers, using 4096 has proven problematic with even a small number of yarn local dirs/disks (3 in the case I saw). We're going to need to make the size of this buffer configurable. If it seems more appropriate to do so in a separate issue, I'm fine with that. Image can be specified by a digest, which is more secure. I do not see that supported by the regex. IMAGE [:TAG|@DIGEST] This is the same regex used on the java side, just duplicated in c-e to address the concerns with direct invocation. We should fix this both places to support the digest notation. Again, this seems like a separate issue from what this patch is addressing.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 0s Docker mode activated.
          -1 patch 0m 6s YARN-6623 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help.



          Subsystem Report/Notes
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882875/YARN-6623.004.patch
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/17031/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 0s Docker mode activated. -1 patch 0m 6s YARN-6623 does not apply to trunk. Rebase required? Wrong Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. Subsystem Report/Notes JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882875/YARN-6623.004.patch Console output https://builds.apache.org/job/PreCommit-YARN-Build/17031/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Thanks for the review Miklos Szegedi!

          /proc/mounts,/sys/fs/cgroup are always in the same place

          This is actually not completely true. If you run in a container,/sys/fs/cgroup can be anywhere

          490 .addReadOnlyMountLocation(CGROUPS_ROOT_DIRECTORY,
          491 CGROUPS_ROOT_DIRECTORY, false);

          This is actually a security issue. As opposed to lxcfs, mounting cgroups will expose information about all the other containers to each container. This change is big enough but you might want to whitelist this in the future.

          This was already being set before this patch. Perhaps, it can be handled cleanly as part of YARN-5534?

          SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -std=c++11")

          Not sure, but this might not build on old OS like centos6

          Fixed.

          76 printWriter.println("[docker-command-execution]");
          77 for (Map.Entry<String, List<String>> entry : cmd.getCommandArguments()
          78 .entrySet()) {
          79 printWriter.println(" " + entry.getKey() + "=" + StringUtils
          80 .join(",", entry.getValue()));
          81 }

          Probably it does worth to check, if entry for example contains “abc=\ndef” to avoid injection attacks.

          Fixed.

          169 ret[i + 1] = '\0';

          I think it is safe to do a single ret[I] = 0; after the loop

          Fixed.

          177 void quote_and_append_arg(char *str, size_t *size, const char param, const char *arg) {
          178 char *tmp = escape_single_quote(arg);
          179 strcat(*str, param);
          180 strcat(*str, "'");
          181 if(strlen(*str) + strlen(tmp) > *size) {
          182 *str = (char *) realloc(*str, strlen(*str) + strlen(tmp) + 1024);
          183 if(*str == NULL) {
          184 exit(OUT_OF_MEMORY);
          185 }
          186 *size = strlen(*str) + strlen(tmp) + 1024;
          187 }
          188 strcat(*str, tmp);
          189 strcat(*str, "' ");
          190 free(tmp);
          191 }

          What is 1024? How about setting * size before *str, so that there is no need for duplication?

          Fixed.

          #define EXECUTOR_PATH_MAX 131072

          This is lots of allocation. The OS actually needs to zero all the allocated memory before giving it to other processes after close, so this might add to the overall CPU usage and memory bandwidth.

          Fixed. Set back to 4096.

          35 if (command != NULL) {
          36 free(command);
          37 }

          This is not necessary, free always ignores NULL argument.

          Fixed.

          47 if (current_len + string_len < bufflen) {

          bufflen-1 to allocate space for the terminating 0

          Fixed.

          48 strncpy(buff + current_len, string, bufflen - current_len);

          strncpy does not add 0 terminator at the end of the target, if strlen(string)==bufflen - current_len resulting in a read buffer overflow later.

          Fixed.

          165 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX, strlen(CONTAINER_NAME_PREFIX))) {

          This is a double scan. You can just use strcmp with the same effect.

          249 const char *regex_str = "^(([a-zA-Z0-9.])(:[0-9])?/)?([a-z0-9_./])(:[a-zA-Z0-9_.-])?$”;

          Image can be specified by a digest, which is more secure. I do not see that supported by the regex. IMAGE[:TAG|@DIGEST]

          Shane Kumpf added this, I'm just moving it to a different file. Shane can you please comment?

          1. docker.binary=/bin/docker
            115 docker_binary = strdup("/usr/bin/docker”);

          We should choose one and use it everywhere.

          Fixed.

          477 permitted_mount_len = strlen(permitted_mounts[i]);
          478 if (permitted_mounts[i][permitted_mount_len - 1] == '/') {

          Buffer underflow at permitted_mount_len == 0

          Fixed.

          488 static char* normalize_mount(const char* mount) {

          It would be really nice to have some comments here.

          Fixed.

          503 size_t len = strlen(real_mount);
          504 if (real_mount[len - 1] != '/') {
          505 ret_ptr = (char *) alloc_and_clear_memory(len + 2, sizeof(char));
          506 strncpy(ret_ptr, real_mount, len);
          507 ret_ptr[len] = '/';
          508 ret_ptr[len + 1] = '\0';

          Potential buffer underflow moreover most likely the character does not match and we end with a normalized mount path of “/“. This happens, when strlen(real_mount)==0

          Fixed

          Show
          vvasudev Varun Vasudev added a comment - Thanks for the review Miklos Szegedi ! /proc/mounts,/sys/fs/cgroup are always in the same place This is actually not completely true. If you run in a container,/sys/fs/cgroup can be anywhere 490 .addReadOnlyMountLocation(CGROUPS_ROOT_DIRECTORY, 491 CGROUPS_ROOT_DIRECTORY, false); This is actually a security issue. As opposed to lxcfs, mounting cgroups will expose information about all the other containers to each container. This change is big enough but you might want to whitelist this in the future. This was already being set before this patch. Perhaps, it can be handled cleanly as part of YARN-5534 ? SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -std=c++11") Not sure, but this might not build on old OS like centos6 Fixed. 76 printWriter.println("[docker-command-execution]"); 77 for (Map.Entry<String, List<String>> entry : cmd.getCommandArguments() 78 .entrySet()) { 79 printWriter.println(" " + entry.getKey() + "=" + StringUtils 80 .join(",", entry.getValue())); 81 } Probably it does worth to check, if entry for example contains “abc=\ndef” to avoid injection attacks. Fixed. 169 ret[i + 1] = '\0'; I think it is safe to do a single ret[I] = 0; after the loop Fixed. 177 void quote_and_append_arg(char * str, size_t *size, const char param, const char *arg) { 178 char *tmp = escape_single_quote(arg); 179 strcat(*str, param); 180 strcat(*str, "'"); 181 if(strlen(*str) + strlen(tmp) > *size) { 182 *str = (char *) realloc(*str, strlen(*str) + strlen(tmp) + 1024); 183 if(*str == NULL) { 184 exit(OUT_OF_MEMORY); 185 } 186 *size = strlen(*str) + strlen(tmp) + 1024; 187 } 188 strcat(*str, tmp); 189 strcat(*str, "' "); 190 free(tmp); 191 } What is 1024? How about setting * size before *str, so that there is no need for duplication? Fixed. #define EXECUTOR_PATH_MAX 131072 This is lots of allocation. The OS actually needs to zero all the allocated memory before giving it to other processes after close, so this might add to the overall CPU usage and memory bandwidth. Fixed. Set back to 4096. 35 if (command != NULL) { 36 free(command); 37 } This is not necessary, free always ignores NULL argument. Fixed. 47 if (current_len + string_len < bufflen) { bufflen-1 to allocate space for the terminating 0 Fixed. 48 strncpy(buff + current_len, string, bufflen - current_len); strncpy does not add 0 terminator at the end of the target, if strlen(string)==bufflen - current_len resulting in a read buffer overflow later. Fixed. 165 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX, strlen(CONTAINER_NAME_PREFIX))) { This is a double scan. You can just use strcmp with the same effect. 249 const char *regex_str = "^(([a-zA-Z0-9. ] )(:[0-9] )?/)?([a-z0-9_./ ] )(:[a-zA-Z0-9_.-] )?$”; Image can be specified by a digest, which is more secure. I do not see that supported by the regex. IMAGE[:TAG|@DIGEST] Shane Kumpf added this, I'm just moving it to a different file. Shane can you please comment? docker.binary=/bin/docker 115 docker_binary = strdup("/usr/bin/docker”); We should choose one and use it everywhere. Fixed. 477 permitted_mount_len = strlen(permitted_mounts[i]); 478 if (permitted_mounts[i][permitted_mount_len - 1] == '/') { Buffer underflow at permitted_mount_len == 0 Fixed. 488 static char* normalize_mount(const char* mount) { It would be really nice to have some comments here. Fixed. 503 size_t len = strlen(real_mount); 504 if (real_mount[len - 1] != '/') { 505 ret_ptr = (char *) alloc_and_clear_memory(len + 2, sizeof(char)); 506 strncpy(ret_ptr, real_mount, len); 507 ret_ptr[len] = '/'; 508 ret_ptr[len + 1] = '\0'; Potential buffer underflow moreover most likely the character does not match and we end with a normalized mount path of “/“. This happens, when strlen(real_mount)==0 Fixed
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - - edited

          Thank you for the patch, Varun Vasudev. I had time to check only the first half so far. I hope this helps.

          /proc/mounts,/sys/fs/cgroup are always in the same place
          

          This is actually not completely true. If you run in a container,/sys/fs/cgroup can be anywhere

          SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -std=c++11")
          

          Not sure, but this might not build on old OS like centos6

          490            .addReadOnlyMountLocation(CGROUPS_ROOT_DIRECTORY,
          491                CGROUPS_ROOT_DIRECTORY, false);
          

          This is actually a security issue. As opposed to lxcfs, mounting cgroups will expose information about all the other containers to each container. This change is big enough but you might want to whitelist this in the future.

          76          printWriter.println("[docker-command-execution]");
          77          for (Map.Entry<String, List<String>> entry : cmd.getCommandArguments()
          78              .entrySet()) {
          79            printWriter.println("  " + entry.getKey() + "=" + StringUtils
          80                .join(",", entry.getValue()));
          81          }
          

          Probably it does worth to check, if entry for example contains “abc=\ndef” to avoid injection attacks.

          169          ret[i + 1] = '\0';
          

          I think it is safe to do a single ret[I] = 0; after the loop

          177    void quote_and_append_arg(char **str, size_t *size, const char* param, const char *arg) {
          178      char *tmp = escape_single_quote(arg);
          179      strcat(*str, param);
          180      strcat(*str, "'");
          181      if(strlen(*str) + strlen(tmp) > *size) {
          182        *str = (char *) realloc(*str, strlen(*str) + strlen(tmp) + 1024);
          183        if(*str == NULL) {
          184          exit(OUT_OF_MEMORY);
          185        }
          186        *size = strlen(*str) + strlen(tmp) + 1024;
          187      }
          188      strcat(*str, tmp);
          189      strcat(*str, "' ");
          190      free(tmp);
          191    }
          

          What is 1024? How about setting * size before *str, so that there is no need for duplication?

          #define EXECUTOR_PATH_MAX 131072
          

          This is lots of allocation. The OS actually needs to zero all the allocated memory before giving it to other processes after close, so this might add to the overall CPU usage and memory bandwidth.

          35        if (command != NULL) {
          36          free(command);
          37        }
          

          This is not necessary, free always ignores NULL argument.

          47      if (current_len + string_len < bufflen) {
          

          bufflen-1 to allocate space for the terminating 0

          48        strncpy(buff + current_len, string, bufflen - current_len);
          

          strncpy does not add 0 terminator at the end of the target, if strlen(string)==bufflen - current_len resulting in a read buffer overflow later.

          #  docker.binary=/bin/docker
          115          docker_binary = strdup("/usr/bin/docker”);
          

          We should choose one and use it everywhere.

          165      if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX, strlen(CONTAINER_NAME_PREFIX))) {
          

          This is a double scan. You can just use strcmp with the same effect.

          249      const char *regex_str = "^(([a-zA-Z0-9.-]+)(:[0-9]+)?/)?([a-z0-9_./-]+)(:[a-zA-Z0-9_.-]+)?$”;
          

          Image can be specified by a digest, which is more secure. I do not see that supported by the regex. IMAGE[:TAG|@DIGEST]

          477        permitted_mount_len = strlen(permitted_mounts[i]);
          478        if (permitted_mounts[i][permitted_mount_len - 1] == '/') {
          

          Buffer underflow at permitted_mount_len == 0

          488    static char* normalize_mount(const char* mount) {
          

          It would be really nice to have some comments here.

          503          size_t len = strlen(real_mount);
          504          if (real_mount[len - 1] != '/') {
          505            ret_ptr = (char *) alloc_and_clear_memory(len + 2, sizeof(char));
          506            strncpy(ret_ptr, real_mount, len);
          507            ret_ptr[len] = '/';
          508            ret_ptr[len + 1] = '\0';
          

          Potential buffer underflow moreover most likely the character does not match and we end with a normalized mount path of “/“. This happens, when strlen(real_mount)==0
          continued...

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - - edited Thank you for the patch, Varun Vasudev . I had time to check only the first half so far. I hope this helps. /proc/mounts,/sys/fs/cgroup are always in the same place This is actually not completely true. If you run in a container,/sys/fs/cgroup can be anywhere SET(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -Wall -std=c++11" ) Not sure, but this might not build on old OS like centos6 490 .addReadOnlyMountLocation(CGROUPS_ROOT_DIRECTORY, 491 CGROUPS_ROOT_DIRECTORY, false ); This is actually a security issue. As opposed to lxcfs, mounting cgroups will expose information about all the other containers to each container. This change is big enough but you might want to whitelist this in the future. 76 printWriter.println( "[docker-command-execution]" ); 77 for (Map.Entry< String , List< String >> entry : cmd.getCommandArguments() 78 .entrySet()) { 79 printWriter.println( " " + entry.getKey() + "=" + StringUtils 80 .join( "," , entry.getValue())); 81 } Probably it does worth to check, if entry for example contains “abc=\ndef” to avoid injection attacks. 169 ret[i + 1] = '\0'; I think it is safe to do a single ret [I] = 0; after the loop 177 void quote_and_append_arg( char **str, size_t *size, const char * param, const char *arg) { 178 char *tmp = escape_single_quote(arg); 179 strcat(*str, param); 180 strcat(*str, "'" ); 181 if (strlen(*str) + strlen(tmp) > *size) { 182 *str = ( char *) realloc(*str, strlen(*str) + strlen(tmp) + 1024); 183 if (*str == NULL) { 184 exit(OUT_OF_MEMORY); 185 } 186 *size = strlen(*str) + strlen(tmp) + 1024; 187 } 188 strcat(*str, tmp); 189 strcat(*str, "' " ); 190 free(tmp); 191 } What is 1024? How about setting * size before *str, so that there is no need for duplication? #define EXECUTOR_PATH_MAX 131072 This is lots of allocation. The OS actually needs to zero all the allocated memory before giving it to other processes after close, so this might add to the overall CPU usage and memory bandwidth. 35 if (command != NULL) { 36 free(command); 37 } This is not necessary, free always ignores NULL argument. 47 if (current_len + string_len < bufflen) { bufflen-1 to allocate space for the terminating 0 48 strncpy(buff + current_len, string, bufflen - current_len); strncpy does not add 0 terminator at the end of the target, if strlen(string)==bufflen - current_len resulting in a read buffer overflow later. # docker.binary=/bin/docker 115 docker_binary = strdup("/usr/bin/docker”); We should choose one and use it everywhere. 165 if (0 == strncmp(container_name, CONTAINER_NAME_PREFIX, strlen(CONTAINER_NAME_PREFIX))) { This is a double scan. You can just use strcmp with the same effect. 249 const char *regex_str = "^(([a-zA-Z0-9.-]+)(:[0-9]+)?/)?([a-z0-9_./-]+)(:[a-zA-Z0-9_.-]+)?$”; Image can be specified by a digest, which is more secure. I do not see that supported by the regex. IMAGE [:TAG|@DIGEST] 477 permitted_mount_len = strlen(permitted_mounts[i]); 478 if (permitted_mounts[i][permitted_mount_len - 1] == '/') { Buffer underflow at permitted_mount_len == 0 488 static char * normalize_mount( const char * mount) { It would be really nice to have some comments here. 503 size_t len = strlen(real_mount); 504 if (real_mount[len - 1] != '/') { 505 ret_ptr = ( char *) alloc_and_clear_memory(len + 2, sizeof( char )); 506 strncpy(ret_ptr, real_mount, len); 507 ret_ptr[len] = '/'; 508 ret_ptr[len + 1] = '\0'; Potential buffer underflow moreover most likely the character does not match and we end with a normalized mount path of “/“. This happens, when strlen(real_mount)==0 continued...
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 16s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 49s Maven dependency ordering for branch
          +1 mvninstall 17m 36s trunk passed
          +1 compile 11m 19s trunk passed
          +1 checkstyle 1m 7s trunk passed
          +1 mvnsite 4m 39s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 1m 1s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 47s trunk passed
                Patch Compile Tests
          0 mvndep 0m 13s Maven dependency ordering for patch
          +1 mvninstall 4m 35s the patch passed
          +1 compile 7m 40s the patch passed
          +1 cc 7m 40s the patch passed
          +1 javac 7m 40s the patch passed
          +1 checkstyle 1m 9s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 25 unchanged - 2 fixed = 25 total (was 27)
          +1 mvnsite 5m 20s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 2s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          +1 findbugs 1m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          +1 javadoc 2m 51s the patch passed
                Other Tests
          -1 unit 77m 57s hadoop-yarn in the patch failed.
          +1 unit 14m 36s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 34s The patch does not generate ASF License warnings.
          163m 19s



          Reason Tests
          Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
          Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882087/YARN-6623.003.patch
          Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle
          uname Linux b1c5b3058eca 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 588c190
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16926/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16926/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16926/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16926/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 16s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 49s Maven dependency ordering for branch +1 mvninstall 17m 36s trunk passed +1 compile 11m 19s trunk passed +1 checkstyle 1m 7s trunk passed +1 mvnsite 4m 39s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 1m 1s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 47s trunk passed       Patch Compile Tests 0 mvndep 0m 13s Maven dependency ordering for patch +1 mvninstall 4m 35s the patch passed +1 compile 7m 40s the patch passed +1 cc 7m 40s the patch passed +1 javac 7m 40s the patch passed +1 checkstyle 1m 9s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 25 unchanged - 2 fixed = 25 total (was 27) +1 mvnsite 5m 20s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 2s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 1m 18s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) +1 javadoc 2m 51s the patch passed       Other Tests -1 unit 77m 57s hadoop-yarn in the patch failed. +1 unit 14m 36s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 34s The patch does not generate ASF License warnings. 163m 19s Reason Tests Failed junit tests hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation Timed out junit tests org.apache.hadoop.yarn.server.resourcemanager.recovery.TestZKRMStateStore Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12882087/YARN-6623.003.patch Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle uname Linux b1c5b3058eca 3.13.0-116-generic #163-Ubuntu SMP Fri Mar 31 14:13:22 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 588c190 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16926/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16926/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16926/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16926/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded 003.patch which fixes the failing cetest tests and has some checkstyle fixes.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded 003.patch which fixes the failing cetest tests and has some checkstyle fixes.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 17s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 55s Maven dependency ordering for branch
          +1 mvninstall 14m 19s trunk passed
          +1 compile 9m 51s trunk passed
          +1 checkstyle 0m 57s trunk passed
          +1 mvnsite 3m 31s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 4s trunk passed
                Patch Compile Tests
          0 mvndep 0m 11s Maven dependency ordering for patch
          +1 mvninstall 3m 11s the patch passed
          +1 compile 5m 41s the patch passed
          +1 cc 5m 41s the patch passed
          +1 javac 5m 41s the patch passed
          -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 25 unchanged - 2 fixed = 31 total (was 27)
          +1 mvnsite 3m 27s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          +1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1)
          +1 javadoc 1m 57s the patch passed
                Other Tests
          -1 unit 72m 12s hadoop-yarn in the patch failed.
          -1 unit 14m 30s hadoop-yarn-server-nodemanager in the patch failed.
          +1 asflicense 0m 33s The patch does not generate ASF License warnings.
          143m 21s



          Reason Tests
          Failed junit tests TEST-cetest
            hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation
            TEST-cetest



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881942/YARN-6623.002.patch
          Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle
          uname Linux da53d541ddef 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / 2e43c28
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16908/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16908/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16908/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16908/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16908/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16908/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 17s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 55s Maven dependency ordering for branch +1 mvninstall 14m 19s trunk passed +1 compile 9m 51s trunk passed +1 checkstyle 0m 57s trunk passed +1 mvnsite 3m 31s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 49s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 4s trunk passed       Patch Compile Tests 0 mvndep 0m 11s Maven dependency ordering for patch +1 mvninstall 3m 11s the patch passed +1 compile 5m 41s the patch passed +1 cc 5m 41s the patch passed +1 javac 5m 41s the patch passed -0 checkstyle 0m 56s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 25 unchanged - 2 fixed = 31 total (was 27) +1 mvnsite 3m 27s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn +1 findbugs 0m 52s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 0 new + 0 unchanged - 1 fixed = 0 total (was 1) +1 javadoc 1m 57s the patch passed       Other Tests -1 unit 72m 12s hadoop-yarn in the patch failed. -1 unit 14m 30s hadoop-yarn-server-nodemanager in the patch failed. +1 asflicense 0m 33s The patch does not generate ASF License warnings. 143m 21s Reason Tests Failed junit tests TEST-cetest   hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation   TEST-cetest Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881942/YARN-6623.002.patch Optional Tests asflicense findbugs xml compile cc mvnsite javac unit javadoc mvninstall checkstyle uname Linux da53d541ddef 3.13.0-119-generic #166-Ubuntu SMP Wed May 3 12:18:55 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / 2e43c28 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16908/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16908/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16908/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16908/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16908/testReport/ modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16908/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          ebadger Eric Badger added a comment -

          I'll review the patch sometime this week

          Show
          ebadger Eric Badger added a comment - I'll review the patch sometime this week
          Hide
          vvasudev Varun Vasudev added a comment -

          Uploaded a new patch to fix the test failures and findbugs warnings.

          Show
          vvasudev Varun Vasudev added a comment - Uploaded a new patch to fix the test failures and findbugs warnings.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 14s Docker mode activated.
                Prechecks
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.
                trunk Compile Tests
          0 mvndep 0m 53s Maven dependency ordering for branch
          +1 mvninstall 13m 38s trunk passed
          +1 compile 9m 22s trunk passed
          +1 checkstyle 0m 50s trunk passed
          +1 mvnsite 3m 33s trunk passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 0m 45s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings.
          +1 javadoc 2m 0s trunk passed
                Patch Compile Tests
          0 mvndep 0m 8s Maven dependency ordering for patch
          +1 mvninstall 3m 6s the patch passed
          +1 compile 6m 37s the patch passed
          +1 cc 6m 37s the patch passed
          +1 javac 6m 37s the patch passed
          -0 checkstyle 0m 54s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 25 unchanged - 2 fixed = 31 total (was 27)
          +1 mvnsite 4m 0s the patch passed
          -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn
          -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1)
          +1 javadoc 2m 18s the patch passed
                Other Tests
          -1 unit 75m 59s hadoop-yarn in the patch failed.
          -1 unit 13m 25s hadoop-yarn-server-nodemanager in the patch failed.
          -1 asflicense 0m 25s The patch generated 1 ASF License warnings.
          146m 16s



          Reason Tests
          FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
            org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.docker.DockerCommand.toString() concatenates strings using + in a loop At DockerCommand.java:in a loop At DockerCommand.java:[line 87]
            Unread field:field be static? At DockerCommand.java:[line 43]
          Failed junit tests hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime
            hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps
            hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities
            hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:14b5c93
          JIRA Issue YARN-6623
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881782/YARN-6623.001.patch
          Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle
          uname Linux 711bfbad4781 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / d8f74c3
          Default Java 1.8.0_144
          findbugs v3.1.0-RC1
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html
          checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt
          whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/whitespace-eol.txt
          findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16887/testReport/
          asflicense https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/patch-asflicense-problems.txt
          modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/16887/console
          Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 14s Docker mode activated.       Prechecks +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 13 new or modified test files.       trunk Compile Tests 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 13m 38s trunk passed +1 compile 9m 22s trunk passed +1 checkstyle 0m 50s trunk passed +1 mvnsite 3m 33s trunk passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 0m 45s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager in trunk has 1 extant Findbugs warnings. +1 javadoc 2m 0s trunk passed       Patch Compile Tests 0 mvndep 0m 8s Maven dependency ordering for patch +1 mvninstall 3m 6s the patch passed +1 compile 6m 37s the patch passed +1 cc 6m 37s the patch passed +1 javac 6m 37s the patch passed -0 checkstyle 0m 54s hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 25 unchanged - 2 fixed = 31 total (was 27) +1 mvnsite 4m 0s the patch passed -1 whitespace 0m 0s The patch has 1 line(s) that end in whitespace. Use git apply --whitespace=fix <<patch_file>>. Refer https://git-scm.com/docs/git-apply 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn -1 findbugs 1m 5s hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager generated 2 new + 1 unchanged - 0 fixed = 3 total (was 1) +1 javadoc 2m 18s the patch passed       Other Tests -1 unit 75m 59s hadoop-yarn in the patch failed. -1 unit 13m 25s hadoop-yarn-server-nodemanager in the patch failed. -1 asflicense 0m 25s The patch generated 1 ASF License warnings. 146m 16s Reason Tests FindBugs module:hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager   org.apache.hadoop.yarn.server.nodemanager.containermanager.linux.runtime.docker.DockerCommand.toString() concatenates strings using + in a loop At DockerCommand.java:in a loop At DockerCommand.java: [line 87]   Unread field:field be static? At DockerCommand.java: [line 43] Failed junit tests hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageApps   hadoop.yarn.server.timelineservice.storage.TestHBaseTimelineStorageEntities   hadoop.yarn.server.nodemanager.containermanager.linux.runtime.TestDockerContainerRuntime Subsystem Report/Notes Docker Image:yetus/hadoop:14b5c93 JIRA Issue YARN-6623 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12881782/YARN-6623.001.patch Optional Tests asflicense compile cc mvnsite javac unit javadoc mvninstall findbugs checkstyle uname Linux 711bfbad4781 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12 13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / d8f74c3 Default Java 1.8.0_144 findbugs v3.1.0-RC1 findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/branch-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-warnings.html checkstyle https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/diff-checkstyle-hadoop-yarn-project_hadoop-yarn.txt whitespace https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/whitespace-eol.txt findbugs https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/new-findbugs-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.html unit https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/16887/testReport/ asflicense https://builds.apache.org/job/PreCommit-YARN-Build/16887/artifact/patchprocess/patch-asflicense-problems.txt modules C: hadoop-yarn-project/hadoop-yarn hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/16887/console Powered by Apache Yetus 0.6.0-SNAPSHOT http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          First cut of the patch. It's big because it moves a bunch of functionality into the container-executor and the subsequent refactor on the java code has a lot of changes.

          At a high level, the changes are -
          1. Move the Docker command construction into the container-executor(docker-util.c)
          2. The NM passes a file with params to the container-executor instead of a file containing the commands.
          3. Added controls in container-executor.cfg to explicitly allow capabilities, mount directories, mount devices, and allowed networks. By default, nothing is allowed.
          4. Added a control in container-executor to disable privileged containers.

          Shane Kumpf, Miklos Szegedi, Eric Badger, Wangda Tan - can you please take a look?

          Thanks!

          Show
          vvasudev Varun Vasudev added a comment - First cut of the patch. It's big because it moves a bunch of functionality into the container-executor and the subsequent refactor on the java code has a lot of changes. At a high level, the changes are - 1. Move the Docker command construction into the container-executor(docker-util.c) 2. The NM passes a file with params to the container-executor instead of a file containing the commands. 3. Added controls in container-executor.cfg to explicitly allow capabilities, mount directories, mount devices, and allowed networks. By default, nothing is allowed. 4. Added a control in container-executor to disable privileged containers. Shane Kumpf , Miklos Szegedi , Eric Badger , Wangda Tan - can you please take a look? Thanks!
          Hide
          vvasudev Varun Vasudev added a comment -

          To fix this, we need support for sections provided by YARN-6033.

          Show
          vvasudev Varun Vasudev added a comment - To fix this, we need support for sections provided by YARN-6033 .
          Hide
          vvasudev Varun Vasudev added a comment -

          Just to note - we should document this in the release notes since it'll break backwards compatibility(if we turn off all runtime except the process runtime by default).

          Show
          vvasudev Varun Vasudev added a comment - Just to note - we should document this in the release notes since it'll break backwards compatibility(if we turn off all runtime except the process runtime by default).
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          I updated this to blocker, because I think 3.0 should not ship without this Jira. Let me know, if you have any concerns.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - I updated this to blocker, because I think 3.0 should not ship without this Jira. Let me know, if you have any concerns.
          Hide
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment -

          Daniel Templeton, this is needed I think for defense in depth. container-executor.cfg is enforced to be runnable only by root. yarn-site.xml is not. Also container-executor does not allow now to launch something impersonating root. This feature should be followed by the Docker code as well.

          /**
           * Is the user a real user account?
           * Checks:
           *   1. Not root
           *   2. UID is above the minimum configured.
           *   3. Not in banned user list
           * Returns NULL on failure
           */
          struct passwd* check_user(const char *user) {
          

          Let's assume someone allows the container-executor executed from yarn but set user to root (or run privileged docker). In this case the point running YARN as yarn and not root is lost.

          Show
          miklos.szegedi@cloudera.com Miklos Szegedi added a comment - Daniel Templeton , this is needed I think for defense in depth. container-executor.cfg is enforced to be runnable only by root. yarn-site.xml is not. Also container-executor does not allow now to launch something impersonating root. This feature should be followed by the Docker code as well. /** * Is the user a real user account? * Checks: * 1. Not root * 2. UID is above the minimum configured. * 3. Not in banned user list * Returns NULL on failure */ struct passwd* check_user( const char *user) { Let's assume someone allows the container-executor executed from yarn but set user to root (or run privileged docker). In this case the point running YARN as yarn and not root is lost.
          Hide
          templedf Daniel Templeton added a comment -

          It's not obvious to me why having two ways to disable privileged containers is necessary. The container-executer.cfg also lives on the NM, so what we're saying is that we want two ways to disable privileged containers on the NM, both controlled by the administrator. Is the point to keep someone from being able to use the container-executor binary as a security exploit outside of the NM? If someone manages to gain the ability to launch the container-executor directly, privileged containers are the least of our worries.

          Show
          templedf Daniel Templeton added a comment - It's not obvious to me why having two ways to disable privileged containers is necessary. The container-executer.cfg also lives on the NM, so what we're saying is that we want two ways to disable privileged containers on the NM, both controlled by the administrator. Is the point to keep someone from being able to use the container-executor binary as a security exploit outside of the NM? If someone manages to gain the ability to launch the container-executor directly, privileged containers are the least of our worries.

            People

            • Assignee:
              vvasudev Varun Vasudev
              Reporter:
              vvasudev Varun Vasudev
            • Votes:
              0 Vote for this issue
              Watchers:
              21 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development