Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5388

Deprecate and remove DockerContainerExecutor

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.9.0, 3.0.0-alpha2
    • Component/s: nodemanager
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Incompatible change, Reviewed
    • Release Note:
      DockerContainerExecutor is deprecated starting 2.9.0 and removed from 3.0.0. Please use LinuxContainerExecutor with the DockerRuntime to run Docker containers on YARN clusters.

      Description

      Because the DockerContainerExecuter overrides the writeLaunchEnv() method, it must also have the wildcard processing logic from YARN-4958/YARN-5373 added to it. Without it, the use of -libjars will fail unless wildcarding is disabled.

      1. YARN-5388.001.patch
        57 kB
        Daniel Templeton
      2. YARN-5388.002.patch
        58 kB
        Daniel Templeton
      3. YARN-5388.003.patch
        68 kB
        Daniel Templeton
      4. YARN-5388.branch-2.001.patch
        4 kB
        Daniel Templeton
      5. YARN-5388.branch-2.002.patch
        5 kB
        Daniel Templeton
      6. YARN-5388.branch-2.003.patch
        6 kB
        Daniel Templeton

        Issue Links

          Activity

          Hide
          kasha Karthik Kambatla added a comment -

          Related-but-different question: Sidharta Seethana, Vinod Kumar Vavilapalli - what is the long-term plan for DCE given LCE is supporting Docker? Should we consider deprecating it in 2.8?

          Show
          kasha Karthik Kambatla added a comment - Related-but-different question: Sidharta Seethana , Vinod Kumar Vavilapalli - what is the long-term plan for DCE given LCE is supporting Docker? Should we consider deprecating it in 2.8?
          Hide
          vvasudev Varun Vasudev added a comment -

          I would be in favor of deprecating DCE. I'm not sure where it's being used but it's fairly limited in the use cases it supports.

          Show
          vvasudev Varun Vasudev added a comment - I would be in favor of deprecating DCE. I'm not sure where it's being used but it's fairly limited in the use cases it supports.
          Hide
          kasha Karthik Kambatla added a comment -

          +1 to deprecating DCE in 2.x and removing it altogether in 3.x.

          Show
          kasha Karthik Kambatla added a comment - +1 to deprecating DCE in 2.x and removing it altogether in 3.x.
          Hide
          templedf Daniel Templeton added a comment -

          Here's a patch for branch 2 that deprecates DockerContainerExecutor and fixes what I broke.

          Show
          templedf Daniel Templeton added a comment - Here's a patch for branch 2 that deprecates DockerContainerExecutor and fixes what I broke.
          Hide
          templedf Daniel Templeton added a comment -

          And here's a trunk patch that expunges DockerContainerExecuter from the records.

          Show
          templedf Daniel Templeton added a comment - And here's a trunk patch that expunges DockerContainerExecuter from the records.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 11s Maven dependency ordering for branch
          +1 mvninstall 8m 8s trunk passed
          +1 compile 2m 55s trunk passed
          +1 checkstyle 0m 44s trunk passed
          +1 mvnsite 1m 2s trunk passed
          +1 mvneclipse 0m 27s trunk passed
          +1 findbugs 1m 56s trunk passed
          +1 javadoc 0m 36s trunk passed
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 55s the patch passed
          +1 compile 2m 31s the patch passed
          -1 javac 2m 31s hadoop-yarn-project_hadoop-yarn generated 2 new + 33 unchanged - 2 fixed = 35 total (was 35)
          +1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 206 unchanged - 104 fixed = 206 total (was 310)
          +1 mvnsite 0m 57s the patch passed
          +1 mvneclipse 0m 21s the patch passed
          +1 whitespace 0m 1s The patch has no whitespace issues.
          +1 findbugs 2m 2s the patch passed
          +1 javadoc 0m 15s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 240 unchanged - 2 fixed = 240 total (was 242)
          -1 unit 0m 23s hadoop-yarn-api in the patch failed.
          +1 unit 13m 13s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          39m 4s



          Reason Tests
          Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825926/YARN-5388.001.patch
          JIRA Issue YARN-5388
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux d7e86e0fc01d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c258171
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12919/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
          unit https://builds.apache.org/job/PreCommit-YARN-Build/12919/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12919/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12919/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12919/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 11s Maven dependency ordering for branch +1 mvninstall 8m 8s trunk passed +1 compile 2m 55s trunk passed +1 checkstyle 0m 44s trunk passed +1 mvnsite 1m 2s trunk passed +1 mvneclipse 0m 27s trunk passed +1 findbugs 1m 56s trunk passed +1 javadoc 0m 36s trunk passed 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 55s the patch passed +1 compile 2m 31s the patch passed -1 javac 2m 31s hadoop-yarn-project_hadoop-yarn generated 2 new + 33 unchanged - 2 fixed = 35 total (was 35) +1 checkstyle 0m 41s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 206 unchanged - 104 fixed = 206 total (was 310) +1 mvnsite 0m 57s the patch passed +1 mvneclipse 0m 21s the patch passed +1 whitespace 0m 1s The patch has no whitespace issues. +1 findbugs 2m 2s the patch passed +1 javadoc 0m 15s hadoop-yarn-api in the patch passed. +1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 240 unchanged - 2 fixed = 240 total (was 242) -1 unit 0m 23s hadoop-yarn-api in the patch failed. +1 unit 13m 13s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 39m 4s Reason Tests Failed junit tests hadoop.yarn.conf.TestYarnConfigurationFields Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825926/YARN-5388.001.patch JIRA Issue YARN-5388 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux d7e86e0fc01d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12919/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt unit https://builds.apache.org/job/PreCommit-YARN-Build/12919/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt unit test logs https://builds.apache.org/job/PreCommit-YARN-Build/12919/artifact/patchprocess/patch-unit-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-api.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12919/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12919/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Whoops. Forgot about yarn-default.xml.

          Show
          templedf Daniel Templeton added a comment - Whoops. Forgot about yarn-default.xml.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 20s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          +1 test4tests 0m 0s The patch appears to include 2 new or modified test files.
          0 mvndep 0m 53s Maven dependency ordering for branch
          +1 mvninstall 6m 39s trunk passed
          +1 compile 2m 16s trunk passed
          +1 checkstyle 0m 41s trunk passed
          +1 mvnsite 1m 24s trunk passed
          +1 mvneclipse 0m 39s trunk passed
          +1 findbugs 2m 34s trunk passed
          +1 javadoc 1m 1s trunk passed
          0 mvndep 0m 9s Maven dependency ordering for patch
          +1 mvninstall 1m 8s the patch passed
          +1 compile 2m 13s the patch passed
          -1 javac 2m 13s hadoop-yarn-project_hadoop-yarn generated 2 new + 33 unchanged - 2 fixed = 35 total (was 35)
          +1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 207 unchanged - 104 fixed = 207 total (was 311)
          +1 mvnsite 1m 17s the patch passed
          +1 mvneclipse 0m 31s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 xml 0m 1s The patch has no ill-formed XML file.
          +1 findbugs 2m 53s the patch passed
          +1 javadoc 0m 15s hadoop-yarn-api in the patch passed.
          +1 javadoc 0m 26s hadoop-yarn-common in the patch passed.
          +1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 240 unchanged - 2 fixed = 240 total (was 242)
          +1 unit 0m 23s hadoop-yarn-api in the patch passed.
          +1 unit 2m 15s hadoop-yarn-common in the patch passed.
          +1 unit 13m 0s hadoop-yarn-server-nodemanager in the patch passed.
          +1 asflicense 0m 17s The patch does not generate ASF License warnings.
          43m 1s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:9560f25
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825933/YARN-5388.002.patch
          JIRA Issue YARN-5388
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml
          uname Linux 100bacc3af00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision trunk / c258171
          Default Java 1.8.0_101
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/12921/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt
          Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12921/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/12921/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 20s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. +1 test4tests 0m 0s The patch appears to include 2 new or modified test files. 0 mvndep 0m 53s Maven dependency ordering for branch +1 mvninstall 6m 39s trunk passed +1 compile 2m 16s trunk passed +1 checkstyle 0m 41s trunk passed +1 mvnsite 1m 24s trunk passed +1 mvneclipse 0m 39s trunk passed +1 findbugs 2m 34s trunk passed +1 javadoc 1m 1s trunk passed 0 mvndep 0m 9s Maven dependency ordering for patch +1 mvninstall 1m 8s the patch passed +1 compile 2m 13s the patch passed -1 javac 2m 13s hadoop-yarn-project_hadoop-yarn generated 2 new + 33 unchanged - 2 fixed = 35 total (was 35) +1 checkstyle 0m 37s hadoop-yarn-project/hadoop-yarn: The patch generated 0 new + 207 unchanged - 104 fixed = 207 total (was 311) +1 mvnsite 1m 17s the patch passed +1 mvneclipse 0m 31s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 xml 0m 1s The patch has no ill-formed XML file. +1 findbugs 2m 53s the patch passed +1 javadoc 0m 15s hadoop-yarn-api in the patch passed. +1 javadoc 0m 26s hadoop-yarn-common in the patch passed. +1 javadoc 0m 15s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager generated 0 new + 240 unchanged - 2 fixed = 240 total (was 242) +1 unit 0m 23s hadoop-yarn-api in the patch passed. +1 unit 2m 15s hadoop-yarn-common in the patch passed. +1 unit 13m 0s hadoop-yarn-server-nodemanager in the patch passed. +1 asflicense 0m 17s The patch does not generate ASF License warnings. 43m 1s Subsystem Report/Notes Docker Image:yetus/hadoop:9560f25 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12825933/YARN-5388.002.patch JIRA Issue YARN-5388 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle xml uname Linux 100bacc3af00 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision trunk / c258171 Default Java 1.8.0_101 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/12921/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn.txt Test Results https://builds.apache.org/job/PreCommit-YARN-Build/12921/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/12921/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          Javac warnings are unrelated.

          Show
          templedf Daniel Templeton added a comment - Javac warnings are unrelated.
          Hide
          templedf Daniel Templeton added a comment -

          Sidharta Seethana, any comments?

          Show
          templedf Daniel Templeton added a comment - Sidharta Seethana , any comments?
          Hide
          templedf Daniel Templeton added a comment -

          Sidharta Seethana, Varun Vasudev, any chance you could give me a review?

          Show
          templedf Daniel Templeton added a comment - Sidharta Seethana , Varun Vasudev , any chance you could give me a review?
          Hide
          kasha Karthik Kambatla added a comment -

          +1 for the trunk patch. Sidharta Seethana, Varun Vasudev - would really appreciate a quick nod from you too.

          branch-2 patch: Shouldn't the deprecation be a java annotation as below instead of being part of the javadoc the current patch has? Also, we should add more details on the reasons for deprecation and suggest alternatives.

          @Deprecated
          public class DockerContainerExecutor extends ContainerExecutor {
          
          Show
          kasha Karthik Kambatla added a comment - +1 for the trunk patch. Sidharta Seethana , Varun Vasudev - would really appreciate a quick nod from you too. branch-2 patch: Shouldn't the deprecation be a java annotation as below instead of being part of the javadoc the current patch has? Also, we should add more details on the reasons for deprecation and suggest alternatives. @Deprecated public class DockerContainerExecutor extends ContainerExecutor {
          Hide
          templedf Daniel Templeton added a comment -

          Here's a branch-2 patch with the @Deprecated tag and an explanation.

          Show
          templedf Daniel Templeton added a comment - Here's a branch-2 patch with the @Deprecated tag and an explanation.
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 0m 19s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          +1 mvninstall 6m 51s branch-2 passed
          +1 compile 0m 28s branch-2 passed with JDK v1.8.0_101
          +1 compile 0m 26s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 0m 20s branch-2 passed
          +1 mvnsite 0m 29s branch-2 passed
          +1 mvneclipse 0m 14s branch-2 passed
          +1 findbugs 0m 54s branch-2 passed
          +1 javadoc 0m 17s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 19s branch-2 passed with JDK v1.7.0_111
          +1 mvninstall 0m 24s the patch passed
          +1 compile 0m 20s the patch passed with JDK v1.8.0_101
          -1 javac 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_101 with JDK v1.8.0_101 generated 5 new + 17 unchanged - 0 fixed = 22 total (was 17)
          +1 compile 0m 24s the patch passed with JDK v1.7.0_111
          -1 javac 0m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_111 with JDK v1.7.0_111 generated 5 new + 19 unchanged - 0 fixed = 24 total (was 19)
          +1 checkstyle 0m 16s the patch passed
          +1 mvnsite 0m 26s the patch passed
          +1 mvneclipse 0m 11s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          +1 findbugs 1m 1s the patch passed
          +1 javadoc 0m 14s the patch passed with JDK v1.8.0_101
          +1 javadoc 0m 16s the patch passed with JDK v1.7.0_111
          +1 unit 14m 48s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_101.
          +1 unit 15m 44s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          46m 4s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831819/YARN-5388.branch-2.002.patch
          JIRA Issue YARN-5388
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 568a2c23acc0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 539c3e4
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13299/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_101.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13299/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13299/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13299/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 0m 19s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 mvninstall 6m 51s branch-2 passed +1 compile 0m 28s branch-2 passed with JDK v1.8.0_101 +1 compile 0m 26s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 20s branch-2 passed +1 mvnsite 0m 29s branch-2 passed +1 mvneclipse 0m 14s branch-2 passed +1 findbugs 0m 54s branch-2 passed +1 javadoc 0m 17s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 19s branch-2 passed with JDK v1.7.0_111 +1 mvninstall 0m 24s the patch passed +1 compile 0m 20s the patch passed with JDK v1.8.0_101 -1 javac 0m 20s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_101 with JDK v1.8.0_101 generated 5 new + 17 unchanged - 0 fixed = 22 total (was 17) +1 compile 0m 24s the patch passed with JDK v1.7.0_111 -1 javac 0m 24s hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_111 with JDK v1.7.0_111 generated 5 new + 19 unchanged - 0 fixed = 24 total (was 19) +1 checkstyle 0m 16s the patch passed +1 mvnsite 0m 26s the patch passed +1 mvneclipse 0m 11s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. +1 findbugs 1m 1s the patch passed +1 javadoc 0m 14s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 16s the patch passed with JDK v1.7.0_111 +1 unit 14m 48s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_101. +1 unit 15m 44s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 46m 4s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12831819/YARN-5388.branch-2.002.patch JIRA Issue YARN-5388 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 568a2c23acc0 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18 21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 539c3e4 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13299/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.8.0_101.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/13299/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-nodemanager-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13299/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager Console output https://builds.apache.org/job/PreCommit-YARN-Build/13299/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          templedf Daniel Templeton added a comment -

          The new javadoc warnings are because the class is now deprecated.

          Show
          templedf Daniel Templeton added a comment - The new javadoc warnings are because the class is now deprecated.
          Hide
          vvasudev Varun Vasudev added a comment -

          I'm in favour of removing DockerContainerExecutor but it should be done via the dev list. We should send out an email to the users and yarn dev lists asking if anyone has objections, etc.

          Show
          vvasudev Varun Vasudev added a comment - I'm in favour of removing DockerContainerExecutor but it should be done via the dev list. We should send out an email to the users and yarn dev lists asking if anyone has objections, etc.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Daniel Templeton Apologies for the late response. Took at look at the trunk patch - I think DockerContainerExecutor.md.vm needs to be removed as well. I also see a reference to DockerContainerExecutor in hadoop-project/src/site/site.xml .

          Show
          sidharta-s Sidharta Seethana added a comment - Daniel Templeton Apologies for the late response. Took at look at the trunk patch - I think DockerContainerExecutor.md.vm needs to be removed as well. I also see a reference to DockerContainerExecutor in hadoop-project/src/site/site.xml .
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Daniel Templeton branch-2 patch : should we update DockerContainerExecutor.md.vm here as well?

          Show
          sidharta-s Sidharta Seethana added a comment - Daniel Templeton branch-2 patch : should we update DockerContainerExecutor.md.vm here as well?
          Hide
          templedf Daniel Templeton added a comment -

          Thanks for the review, Sidharta Seethana. I removed the docs from trunk and slapped a big deprecation warning on the docs in branch-2. Since there aren't yet docs on using Docker with the LCE, it seemed cruel to strip the DCE docs out entirely. (See YARN-5258.)

          Show
          templedf Daniel Templeton added a comment - Thanks for the review, Sidharta Seethana . I removed the docs from trunk and slapped a big deprecation warning on the docs in branch-2. Since there aren't yet docs on using Docker with the LCE, it seemed cruel to strip the DCE docs out entirely. (See YARN-5258 .)
          Hide
          hadoopqa Hadoop QA added a comment -
          -1 overall



          Vote Subsystem Runtime Comment
          0 reexec 17m 11s Docker mode activated.
          +1 @author 0m 0s The patch does not contain any @author tags.
          -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch.
          0 mvndep 0m 36s Maven dependency ordering for branch
          +1 mvninstall 6m 53s branch-2 passed
          +1 compile 2m 3s branch-2 passed with JDK v1.8.0_101
          +1 compile 2m 22s branch-2 passed with JDK v1.7.0_111
          +1 checkstyle 0m 40s branch-2 passed
          +1 mvnsite 0m 42s branch-2 passed
          +1 mvneclipse 0m 24s branch-2 passed
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 0m 56s branch-2 passed
          +1 javadoc 0m 25s branch-2 passed with JDK v1.8.0_101
          +1 javadoc 0m 30s branch-2 passed with JDK v1.7.0_111
          0 mvndep 0m 10s Maven dependency ordering for patch
          +1 mvninstall 0m 34s the patch passed
          +1 compile 2m 1s the patch passed with JDK v1.8.0_101
          -1 javac 2m 1s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_101 with JDK v1.8.0_101 generated 5 new + 35 unchanged - 0 fixed = 40 total (was 35)
          +1 compile 2m 21s the patch passed with JDK v1.7.0_111
          -1 javac 2m 21s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_111 with JDK v1.7.0_111 generated 5 new + 43 unchanged - 0 fixed = 48 total (was 43)
          +1 checkstyle 0m 38s the patch passed
          +1 mvnsite 0m 41s the patch passed
          +1 mvneclipse 0m 22s the patch passed
          +1 whitespace 0m 0s The patch has no whitespace issues.
          0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site
          +1 findbugs 1m 3s the patch passed
          +1 javadoc 0m 21s the patch passed with JDK v1.8.0_101
          +1 javadoc 0m 26s the patch passed with JDK v1.7.0_111
          +1 unit 14m 52s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_101.
          +1 unit 0m 7s hadoop-yarn-site in the patch passed with JDK v1.8.0_101.
          +1 unit 15m 41s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_111.
          +1 unit 0m 8s hadoop-yarn-site in the patch passed with JDK v1.7.0_111.
          +1 asflicense 0m 18s The patch does not generate ASF License warnings.
          73m 33s



          Subsystem Report/Notes
          Docker Image:yetus/hadoop:b59b8b7
          JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833763/YARN-5388.branch-2.003.patch
          JIRA Issue YARN-5388
          Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle
          uname Linux 4ba174b80d40 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
          Build tool maven
          Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh
          git revision branch-2 / 7993fb5
          Default Java 1.7.0_111
          Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111
          findbugs v3.0.0
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13408/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_101.txt
          javac https://builds.apache.org/job/PreCommit-YARN-Build/13408/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_111.txt
          JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13408/testReport/
          modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn
          Console output https://builds.apache.org/job/PreCommit-YARN-Build/13408/console
          Powered by Apache Yetus 0.3.0 http://yetus.apache.org

          This message was automatically generated.

          Show
          hadoopqa Hadoop QA added a comment - -1 overall Vote Subsystem Runtime Comment 0 reexec 17m 11s Docker mode activated. +1 @author 0m 0s The patch does not contain any @author tags. -1 test4tests 0m 0s The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. 0 mvndep 0m 36s Maven dependency ordering for branch +1 mvninstall 6m 53s branch-2 passed +1 compile 2m 3s branch-2 passed with JDK v1.8.0_101 +1 compile 2m 22s branch-2 passed with JDK v1.7.0_111 +1 checkstyle 0m 40s branch-2 passed +1 mvnsite 0m 42s branch-2 passed +1 mvneclipse 0m 24s branch-2 passed 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 0m 56s branch-2 passed +1 javadoc 0m 25s branch-2 passed with JDK v1.8.0_101 +1 javadoc 0m 30s branch-2 passed with JDK v1.7.0_111 0 mvndep 0m 10s Maven dependency ordering for patch +1 mvninstall 0m 34s the patch passed +1 compile 2m 1s the patch passed with JDK v1.8.0_101 -1 javac 2m 1s hadoop-yarn-project_hadoop-yarn-jdk1.8.0_101 with JDK v1.8.0_101 generated 5 new + 35 unchanged - 0 fixed = 40 total (was 35) +1 compile 2m 21s the patch passed with JDK v1.7.0_111 -1 javac 2m 21s hadoop-yarn-project_hadoop-yarn-jdk1.7.0_111 with JDK v1.7.0_111 generated 5 new + 43 unchanged - 0 fixed = 48 total (was 43) +1 checkstyle 0m 38s the patch passed +1 mvnsite 0m 41s the patch passed +1 mvneclipse 0m 22s the patch passed +1 whitespace 0m 0s The patch has no whitespace issues. 0 findbugs 0m 0s Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site +1 findbugs 1m 3s the patch passed +1 javadoc 0m 21s the patch passed with JDK v1.8.0_101 +1 javadoc 0m 26s the patch passed with JDK v1.7.0_111 +1 unit 14m 52s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.8.0_101. +1 unit 0m 7s hadoop-yarn-site in the patch passed with JDK v1.8.0_101. +1 unit 15m 41s hadoop-yarn-server-nodemanager in the patch passed with JDK v1.7.0_111. +1 unit 0m 8s hadoop-yarn-site in the patch passed with JDK v1.7.0_111. +1 asflicense 0m 18s The patch does not generate ASF License warnings. 73m 33s Subsystem Report/Notes Docker Image:yetus/hadoop:b59b8b7 JIRA Patch URL https://issues.apache.org/jira/secure/attachment/12833763/YARN-5388.branch-2.003.patch JIRA Issue YARN-5388 Optional Tests asflicense compile javac javadoc mvninstall mvnsite unit findbugs checkstyle uname Linux 4ba174b80d40 3.13.0-95-generic #142-Ubuntu SMP Fri Aug 12 17:00:09 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux Build tool maven Personality /testptch/hadoop/patchprocess/precommit/personality/provided.sh git revision branch-2 / 7993fb5 Default Java 1.7.0_111 Multi-JDK versions /usr/lib/jvm/java-8-oracle:1.8.0_101 /usr/lib/jvm/java-7-openjdk-amd64:1.7.0_111 findbugs v3.0.0 javac https://builds.apache.org/job/PreCommit-YARN-Build/13408/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.8.0_101.txt javac https://builds.apache.org/job/PreCommit-YARN-Build/13408/artifact/patchprocess/diff-compile-javac-hadoop-yarn-project_hadoop-yarn-jdk1.7.0_111.txt JDK v1.7.0_111 Test Results https://builds.apache.org/job/PreCommit-YARN-Build/13408/testReport/ modules C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site U: hadoop-yarn-project/hadoop-yarn Console output https://builds.apache.org/job/PreCommit-YARN-Build/13408/console Powered by Apache Yetus 0.3.0 http://yetus.apache.org This message was automatically generated.
          Hide
          vvasudev Varun Vasudev added a comment -

          I think we can go ahead with this? There weren't any objections on the mailing list.

          Show
          vvasudev Varun Vasudev added a comment - I think we can go ahead with this? There weren't any objections on the mailing list.
          Hide
          kasha Karthik Kambatla added a comment -

          +1 on the trunk patch.

          For the branch-2 patch, I am not sure we should make the code improvements in DockerContainerExecutor.java.

          Show
          kasha Karthik Kambatla added a comment - +1 on the trunk patch. For the branch-2 patch, I am not sure we should make the code improvements in DockerContainerExecutor.java.
          Hide
          templedf Daniel Templeton added a comment -

          Post MAPREDUCE-6719, the DCE is broken, and it isn't deprecated before this patch. I don't think it's reasonable to say that slapping a deprecation label on something absolves us from fixing it when it's broken. The label says that it will go away in the future, not that it's broken and cannot be used.

          Show
          templedf Daniel Templeton added a comment - Post MAPREDUCE-6719 , the DCE is broken, and it isn't deprecated before this patch. I don't think it's reasonable to say that slapping a deprecation label on something absolves us from fixing it when it's broken. The label says that it will go away in the future, not that it's broken and cannot be used.
          Hide
          vvasudev Varun Vasudev added a comment -

          Agree with Daniel Templeton - let's commit the branch-2 changes as well.

          Show
          vvasudev Varun Vasudev added a comment - Agree with Daniel Templeton - let's commit the branch-2 changes as well.
          Hide
          kasha Karthik Kambatla added a comment -

          My bad. I missed the fix parts of the code in the rest of the unrelated improvements. +1. Checking both in..

          (For the unrelated code readability etc. improvements, I always wonder if we should do it in a separate JIRA to avoid confusing folks looking at pulling this patch in later.)

          Show
          kasha Karthik Kambatla added a comment - My bad. I missed the fix parts of the code in the rest of the unrelated improvements. +1. Checking both in.. (For the unrelated code readability etc. improvements, I always wonder if we should do it in a separate JIRA to avoid confusing folks looking at pulling this patch in later.)
          Hide
          hudson Hudson added a comment -

          SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10677 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10677/)
          YARN-5388. Deprecate and remove DockerContainerExecutor. (Daniel (kasha: rev de6faae97c0937dcd969386b12283d60c22dcb02)

          • (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java
          • (edit) hadoop-project/src/site/site.xml
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainerExecutor.md.vm
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          • (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          Show
          hudson Hudson added a comment - SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #10677 (See https://builds.apache.org/job/Hadoop-trunk-Commit/10677/ ) YARN-5388 . Deprecate and remove DockerContainerExecutor. (Daniel (kasha: rev de6faae97c0937dcd969386b12283d60c22dcb02) (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java (edit) hadoop-project/src/site/site.xml (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/DockerContainerExecutor.md.vm (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          Hide
          kasha Karthik Kambatla added a comment -

          Thanks Daniel for working on this, Sid and Varun for the reviews.

          Committed the respective patches to trunk and branch-2.

          Show
          kasha Karthik Kambatla added a comment - Thanks Daniel for working on this, Sid and Varun for the reviews. Committed the respective patches to trunk and branch-2.

            People

            • Assignee:
              templedf Daniel Templeton
              Reporter:
              templedf Daniel Templeton
            • Votes:
              0 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development