Hadoop YARN
  1. Hadoop YARN
  2. YARN-1964

Create Docker analog of the LinuxContainerExecutor in YARN

    Details

    • Type: New Feature New Feature
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.2.0
    • Fix Version/s: 2.6.0
    • Component/s: None
    • Labels:
      None

      Description

      Docker (https://www.docker.io/) is, increasingly, a very popular container technology.

      In context of YARN, the support for Docker will provide a very elegant solution to allow applications to package their software into a Docker container (entire Linux file system incl. custom versions of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite software environment. This provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine).

      1. YARN-1964.patch
        69 kB
        Abin Shahab
      2. YARN-1964.patch
        68 kB
        Abin Shahab
      3. YARN-1964.patch
        68 kB
        Abin Shahab
      4. YARN-1964.patch
        68 kB
        Abin Shahab
      5. YARN-1964.patch
        63 kB
        Abin Shahab
      6. YARN-1964.patch
        63 kB
        Abin Shahab
      7. YARN-1964.patch
        61 kB
        Abin Shahab
      8. YARN-1964.patch
        59 kB
        Abin Shahab
      9. YARN-1964.patch
        62 kB
        Abin Shahab
      10. YARN-1964.patch
        61 kB
        Abin Shahab
      11. YARN-1964.patch
        60 kB
        Abin Shahab
      12. yarn-1964-branch-2.2.0-docker.patch
        19 kB
        Abin Shahab
      13. yarn-1964-branch-2.2.0-docker.patch
        18 kB
        Abin Shahab
      14. yarn-1964-docker.patch
        27 kB
        Abin Shahab
      15. yarn-1964-docker.patch
        27 kB
        Abin Shahab
      16. yarn-1964-docker.patch
        32 kB
        Abin Shahab
      17. yarn-1964-docker.patch
        18 kB
        Abin Shahab
      18. yarn-1964-docker.patch
        18 kB
        Abin Shahab

        Issue Links

          Activity

          Arun C Murthy created issue -
          Hide
          haosdent added a comment -

          Amazing! Is it possible?

          Show
          haosdent added a comment - Amazing! Is it possible?
          Hide
          jay vyas added a comment -

          This takes us away from the java idiom of packaging apps as jars .... Its a pretty bold step so let me play devils advocate, because I think it might not be the best idea.

          1) Tying yarn's JVM NodeManagers to LCE's adds a new dependency to the YARN stack. adds new building/compiling/maintaining costs.

          2) NodeManagers are being run in LinuxContainers quite commonly. This could lead to containers, launching NM's, which again launch containers. Seems kind of yucky dont you think?

          3) It forces yarn to be aware of linux containers. This might lead to encouraging the creation of application code that doesnt run easily outside of containerized environment. As awesome as LCE's are, most java and bigdata apps running in YARN with the hadoop .

          Show
          jay vyas added a comment - This takes us away from the java idiom of packaging apps as jars .... Its a pretty bold step so let me play devils advocate, because I think it might not be the best idea. 1) Tying yarn's JVM NodeManagers to LCE's adds a new dependency to the YARN stack. adds new building/compiling/maintaining costs. 2) NodeManagers are being run in LinuxContainers quite commonly. This could lead to containers, launching NM's, which again launch containers. Seems kind of yucky dont you think? 3) It forces yarn to be aware of linux containers. This might lead to encouraging the creation of application code that doesnt run easily outside of containerized environment. As awesome as LCE's are, most java and bigdata apps running in YARN with the hadoop .
          Hide
          Tsuyoshi Ozawa added a comment -

          Do you mean we will have DockerContainerExecutor instead of LinuxContainerExecutor?

          Show
          Tsuyoshi Ozawa added a comment - Do you mean we will have DockerContainerExecutor instead of LinuxContainerExecutor?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          YARN already has platform specific plugins like LinuxContainerExecutor, this would just be one more option.

          Show
          Vinod Kumar Vavilapalli added a comment - YARN already has platform specific plugins like LinuxContainerExecutor, this would just be one more option.
          Hide
          jay vyas added a comment -

          Ah, sorry, i thought this was meant to be done at a different part of the stack ... So is this jira is specificaly to create a "DockerContainerExectuor" class? then that would a really good idea, and I'm pretty sure it would be feasible. I guess you'd need to add a few parameters to the core-site

          in core-site.xml

          <name>yarn.nodemanager.container-executor.class</name>
          <value>DockerContainerExecutor</value>
          

          and then maybe have a docker-site.xml

          <name>docker.container.container.impl</name>
          <value>PythonCentOSContainer</value>
          <!-- then developers define their different container implementations below --> 
          <name>docker.container.container1</name>
          <value>PythonCentOSContainer</value>
          <name>docker.container.container2</name>
          <value>MyContainerWithPostgres</value>
          ...
          

          And then somehow localize resources in a docker-ish sort of way so that the containers can see all the task resources properly.... Is that the idea here?

          Show
          jay vyas added a comment - Ah, sorry, i thought this was meant to be done at a different part of the stack ... So is this jira is specificaly to create a "DockerContainerExectuor" class? then that would a really good idea, and I'm pretty sure it would be feasible. I guess you'd need to add a few parameters to the core-site in core-site.xml <name>yarn.nodemanager.container-executor.class</name> <value>DockerContainerExecutor</value> and then maybe have a docker-site.xml <name>docker.container.container.impl</name> <value>PythonCentOSContainer</value> <!-- then developers define their different container implementations below --> <name>docker.container.container1</name> <value>PythonCentOSContainer</value> <name>docker.container.container2</name> <value>MyContainerWithPostgres</value> ... And then somehow localize resources in a docker-ish sort of way so that the containers can see all the task resources properly.... Is that the idea here?
          Hide
          Abin Shahab added a comment -

          I have a patch for a docker container executor, and it takes containers and docker run CLI options. Will post it for review.

          Show
          Abin Shahab added a comment - I have a patch for a docker container executor, and it takes containers and docker run CLI options. Will post it for review.
          Arun C Murthy made changes -
          Field Original Value New Value
          Assignee Arun C Murthy [ acmurthy ]
          Arun C Murthy made changes -
          Assignee Abin Shahab [ ashahab ]
          Hide
          Arun C Murthy added a comment -

          Abin Shahab - That's great to hear, awesome! Thanks for taking this up!

          Show
          Arun C Murthy added a comment - Abin Shahab - That's great to hear, awesome! Thanks for taking this up!
          Hide
          Wangda Tan added a comment -

          It's really a great feature to make users can run/distribute packages to YARN platform quite simple.
          I'm wondering if it good to add a pluggable ContainerExecutor which can be specified by application at runtime. Which means, app#1 can choose LinuxContainerExecutor, app#2 can choose DockerContainerExecutor, etc. Different container executors will do resource isolation/enforcement separately. But they will have similar behavior to the upper layer module (like ContainerLauncher, ContainersMonitor, etc.). Because users have to pack everything to Docker package if we don't have such mechanism.
          Hope to get your thoughts for on this

          Show
          Wangda Tan added a comment - It's really a great feature to make users can run/distribute packages to YARN platform quite simple. I'm wondering if it good to add a pluggable ContainerExecutor which can be specified by application at runtime . Which means, app#1 can choose LinuxContainerExecutor, app#2 can choose DockerContainerExecutor, etc. Different container executors will do resource isolation/enforcement separately. But they will have similar behavior to the upper layer module (like ContainerLauncher, ContainersMonitor, etc.). Because users have to pack everything to Docker package if we don't have such mechanism. Hope to get your thoughts for on this
          Hide
          Junping Du added a comment -

          +1 on specifying container type by app at runtime. Different container types (default, LXC, docker, VM box, etc.) have different semantics on isolation of security, namespace/env, performance, etc. Allowing app to specifying technology of containers can largely broaden YARN's capability to meet heterogenous app's requirement on isolation at the same time. In implementation, I think current ContainerExecutor is already pluggable, but we may not want to have several ContainerExecutor instances running inside of NMs. May be we should only have a generic one (like: command pattern) to consolidate operation details? Anyway, these discussions looks a little diverge from original JIRA topic, may be we should file a separated JIRA to discuss this?

          Show
          Junping Du added a comment - +1 on specifying container type by app at runtime. Different container types (default, LXC, docker, VM box, etc.) have different semantics on isolation of security, namespace/env, performance, etc. Allowing app to specifying technology of containers can largely broaden YARN's capability to meet heterogenous app's requirement on isolation at the same time. In implementation, I think current ContainerExecutor is already pluggable, but we may not want to have several ContainerExecutor instances running inside of NMs. May be we should only have a generic one (like: command pattern) to consolidate operation details? Anyway, these discussions looks a little diverge from original JIRA topic, may be we should file a separated JIRA to discuss this?
          Abin Shahab made changes -
          Status Open [ 1 ] In Progress [ 3 ]
          Hide
          Abin Shahab added a comment -

          This is the first patch. It's only for branch-2.2.0. I will create a separate for trunk.
          This adds a new ContainerExecutor called DockerContainerExecutor.
          The ContainerExecutor takes two yarn-site.xml configuration params:
          1) MR_CONTAINER_NAME: The name of the container. This container must be pulled and available on the nodemanagers.
          2) MR_CONTAINER_ARGS: The arguments to the docker container. This can be used to mount the hadoop client jars on the container.

          Show
          Abin Shahab added a comment - This is the first patch. It's only for branch-2.2.0. I will create a separate for trunk. This adds a new ContainerExecutor called DockerContainerExecutor. The ContainerExecutor takes two yarn-site.xml configuration params: 1) MR_CONTAINER_NAME: The name of the container. This container must be pulled and available on the nodemanagers. 2) MR_CONTAINER_ARGS: The arguments to the docker container. This can be used to mount the hadoop client jars on the container.
          Abin Shahab made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Affects Version/s 2.2.0 [ 12325051 ]
          Hide
          Abin Shahab added a comment -

          Noticed that the patch pulled in some unnecessary changes. Submitting a clean one soon.

          Show
          Abin Shahab added a comment - Noticed that the patch pulled in some unnecessary changes. Submitting a clean one soon.
          Abin Shahab made changes -
          Status Patch Available [ 10002 ] Open [ 1 ]
          Hide
          Abin Shahab added a comment -

          This is the first patch. It's only for branch-2.2.0. I will create a separate for trunk.
          This adds a new ContainerExecutor called DockerContainerExecutor.
          The ContainerExecutor takes two yarn-site.xml configuration params:
          1) MR_CONTAINER_NAME: The name of the container. This container must be pulled and available on the nodemanagers.
          2) MR_CONTAINER_ARGS: The arguments to the docker container. This can be used to mount the hadoop client jars on the container.

          Show
          Abin Shahab added a comment - This is the first patch. It's only for branch-2.2.0. I will create a separate for trunk. This adds a new ContainerExecutor called DockerContainerExecutor. The ContainerExecutor takes two yarn-site.xml configuration params: 1) MR_CONTAINER_NAME: The name of the container. This container must be pulled and available on the nodemanagers. 2) MR_CONTAINER_ARGS: The arguments to the docker container. This can be used to mount the hadoop client jars on the container.
          Abin Shahab made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Hide
          Abin Shahab added a comment -

          This adds a new ContainerExecutor called DockerContainerExecutor.
          The ContainerExecutor takes two yarn-site.xml configuration params:
          1) MR_CONTAINER_NAME: The name of the container. This container must be pulled and available on the nodemanagers.
          2) MR_CONTAINER_ARGS: The arguments to the docker container. This can be used to mount the hadoop client jars on the container.

          Show
          Abin Shahab added a comment - This adds a new ContainerExecutor called DockerContainerExecutor. The ContainerExecutor takes two yarn-site.xml configuration params: 1) MR_CONTAINER_NAME: The name of the container. This container must be pulled and available on the nodemanagers. 2) MR_CONTAINER_ARGS: The arguments to the docker container. This can be used to mount the hadoop client jars on the container.
          Abin Shahab made changes -
          Attachment yarn-1964-branch-2.2.0-docker.patch [ 12641778 ]
          Hide
          Abin Shahab added a comment -

          Wangda Tan, I like your suggestion. The patch that I've posted has a version of the DefaultContainerExecutor that launches Docker. Right now the Executor type and Container names can be changed only on yarn-site.xml. However, I would like to move it down to the job configuration so that we can achieve what've said: For matlab jobs, users would specify the matlab container in their jobconf, for their regular mapred job, they would specify yarn containers.

          Junping Du, I agree with the idea of a generic executor. The LinuxContainerExecutor can be extended to launch docker. However, we have to work out how the resource-isolation management is handled between docker and CgroupsLCEResourcesHandler.

          Thanks for the feedback!

          Show
          Abin Shahab added a comment - Wangda Tan, I like your suggestion. The patch that I've posted has a version of the DefaultContainerExecutor that launches Docker. Right now the Executor type and Container names can be changed only on yarn-site.xml. However, I would like to move it down to the job configuration so that we can achieve what've said: For matlab jobs, users would specify the matlab container in their jobconf, for their regular mapred job, they would specify yarn containers. Junping Du, I agree with the idea of a generic executor. The LinuxContainerExecutor can be extended to launch docker. However, we have to work out how the resource-isolation management is handled between docker and CgroupsLCEResourcesHandler. Thanks for the feedback!
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641778/yarn-1964-branch-2.2.0-docker.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3623//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3623//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641778/yarn-1964-branch-2.2.0-docker.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3623//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3623//console This message is automatically generated.
          Hide
          Wangda Tan added a comment -

          Abin Shahab, Thanks for you patch! As you said,

          However, I would like to move it down to the job configuration so that ...

          I hope you could share more ideas job configuration you mentioned, is it a YARN application specific configuration or it will be a part of NM's configuration? I asked this because currently, we don't have such ability (Please correct me if I was wrong) to specify application-specific config when running a YARN application. We should leave an option to user specify their own configuration to RM or NM, which can help them creating runtime-selectable modules (like select CE at runtime).

          Show
          Wangda Tan added a comment - Abin Shahab , Thanks for you patch! As you said, However, I would like to move it down to the job configuration so that ... I hope you could share more ideas job configuration you mentioned, is it a YARN application specific configuration or it will be a part of NM's configuration? I asked this because currently, we don't have such ability (Please correct me if I was wrong) to specify application-specific config when running a YARN application. We should leave an option to user specify their own configuration to RM or NM, which can help them creating runtime-selectable modules (like select CE at runtime).
          Hide
          Junping Du added a comment -

          I think one feasible way to do this now is to add container type info (optional) in specifying ContainerRequest in application and going on to ResourceRequest when AM ask RM, it also can support cases that AM ask more than one type of containers for different purpose. The interesting thing is when resource scheduler try to schedule RR to nodes, it should not only consider location of resource request but also the container types supported NM (assume a heterogenous env in possible). It add a little complexity here but with getting flexibility.

          Show
          Junping Du added a comment - I think one feasible way to do this now is to add container type info (optional) in specifying ContainerRequest in application and going on to ResourceRequest when AM ask RM, it also can support cases that AM ask more than one type of containers for different purpose. The interesting thing is when resource scheduler try to schedule RR to nodes, it should not only consider location of resource request but also the container types supported NM (assume a heterogenous env in possible). It add a little complexity here but with getting flexibility.
          Hide
          jay vyas added a comment -

          I'd suggest this JIRA be renamed: Is it specifically to create a DockerLinuxContainer ? Or is it broader. After all... Docker itself is a "container" and YARN itself is a container, and YARN can run inside of docker, and Multiple NM's can be launched as docker processes, and so on... so I think the exact task of this JIRA should be clarified.

          Show
          jay vyas added a comment - I'd suggest this JIRA be renamed: Is it specifically to create a DockerLinuxContainer ? Or is it broader. After all... Docker itself is a "container" and YARN itself is a container, and YARN can run inside of docker, and Multiple NM's can be launched as docker processes, and so on... so I think the exact task of this JIRA should be clarified.
          Hide
          Wangda Tan added a comment -

          Junping Du, as you mentioned,

          The interesting thing is when resource scheduler try to schedule RR to nodes, it should not only consider location of resource request but also the container types supported NM (assume a heterogenous env in possible).

          We can use YARN-796 descirbe container types supported by NM, that will not need change ApplicationMasterProtocol for supporting different containers. More generically, I think we can add a configuration (org.apache.hadoop.conf.Configuration) to container launch request, and it will be consumed by CE.launchContainer call. Which IMO can get more extensibility for users add more container specific parameters to launch container. Any thoughts?

          Show
          Wangda Tan added a comment - Junping Du , as you mentioned, The interesting thing is when resource scheduler try to schedule RR to nodes, it should not only consider location of resource request but also the container types supported NM (assume a heterogenous env in possible). We can use YARN-796 descirbe container types supported by NM, that will not need change ApplicationMasterProtocol for supporting different containers. More generically, I think we can add a configuration (org.apache.hadoop.conf.Configuration) to container launch request, and it will be consumed by CE.launchContainer call. Which IMO can get more extensibility for users add more container specific parameters to launch container. Any thoughts?
          Hide
          Abin Shahab added a comment -

          Trunk patch.

          Show
          Abin Shahab added a comment - Trunk patch.
          Abin Shahab made changes -
          Attachment yarn-1964-docker.patch [ 12641805 ]
          Hide
          Wangda Tan added a comment -

          Hi jay vyas, I guess what the user scenario this JIRA tracked should be the previous one, for the second one, it's more like packaging/deploying tasks.

          Show
          Wangda Tan added a comment - Hi jay vyas , I guess what the user scenario this JIRA tracked should be the previous one, for the second one, it's more like packaging/deploying tasks.
          Hide
          Abin Shahab added a comment -

          Jay Vyas,
          I think the task here is to enable yarn to launch docker containers, not the other way around. I know it's possible to launch yarn from inside a docker container, but that's outside the scope of this task.

          Show
          Abin Shahab added a comment - Jay Vyas, I think the task here is to enable yarn to launch docker containers, not the other way around. I know it's possible to launch yarn from inside a docker container, but that's outside the scope of this task.
          jay vyas made changes -
          Summary Support Docker containers in YARN Create Docker analog of the LinuxContainerExecutor in YARN
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641805/yarn-1964-docker.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3627//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3627//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641805/yarn-1964-docker.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.TestContainerLaunch +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3627//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3627//console This message is automatically generated.
          Hide
          Junping Du added a comment -

          I'd suggest this JIRA be renamed: Is it specifically to create a DockerLinuxContainer ? Or is it broader. After all... Docker itself is a "container" and YARN itself is a container, and YARN can run inside of docker, and Multiple NM's can be launched as docker processes, and so on... so I think the exact task of this JIRA should be clarified.

          Agree with Abin Shahab that we should make this JIRA's scope focus on Docker implementation. For more broader topics on supporting heterogenous containers in runtime, Wangda Tan, jay vyas, how about we go further discussion in a separated JIRA? - I just filed YARN-1983 for that.

          Show
          Junping Du added a comment - I'd suggest this JIRA be renamed: Is it specifically to create a DockerLinuxContainer ? Or is it broader. After all... Docker itself is a "container" and YARN itself is a container, and YARN can run inside of docker, and Multiple NM's can be launched as docker processes, and so on... so I think the exact task of this JIRA should be clarified. Agree with Abin Shahab that we should make this JIRA's scope focus on Docker implementation. For more broader topics on supporting heterogenous containers in runtime, Wangda Tan , jay vyas , how about we go further discussion in a separated JIRA? - I just filed YARN-1983 for that.
          Hide
          Abin Shahab added a comment -

          This should pass on the branch.

          Show
          Abin Shahab added a comment - This should pass on the branch.
          Abin Shahab made changes -
          Attachment yarn-1964-branch-2.2.0-docker.patch [ 12641977 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641977/yarn-1964-branch-2.2.0-docker.patch
          against trunk revision .

          -1 patch. The patch command could not apply the patch.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3631//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641977/yarn-1964-branch-2.2.0-docker.patch against trunk revision . -1 patch . The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3631//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          Trunk-patch with test passing.

          Show
          Abin Shahab added a comment - Trunk-patch with test passing.
          Abin Shahab made changes -
          Attachment yarn-1964-docker.patch [ 12641983 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12641983/yarn-1964-docker.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          -1 tests included. The patch doesn't appear to include any new or modified tests.
          Please justify why no new tests are needed for this patch.
          Also please list what manual steps were performed to verify this patch.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3632//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3632//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12641983/yarn-1964-docker.patch against trunk revision . +1 @author . The patch does not contain any @author tags. -1 tests included . The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3632//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3632//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          Patch with tests.

          Show
          Abin Shahab added a comment - Patch with tests.
          Abin Shahab made changes -
          Attachment yarn-1964-docker.patch [ 12642187 ]
          Hide
          Abin Shahab added a comment -

          Fixed tests.

          Show
          Abin Shahab added a comment - Fixed tests.
          Abin Shahab made changes -
          Attachment yarn-1964-docker.patch [ 12642188 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642187/yarn-1964-docker.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1295 javac compiler warnings (more than the trunk's current 1278 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.nodemanager.TestDockerContainerExecutorWithMocks
          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3635//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/3635//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3635//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642187/yarn-1964-docker.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1295 javac compiler warnings (more than the trunk's current 1278 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.nodemanager.TestDockerContainerExecutorWithMocks org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3635//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/3635//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3635//console This message is automatically generated.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642188/yarn-1964-docker.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          -1 javac. The applied patch generated 1281 javac compiler warnings (more than the trunk's current 1278 warnings).

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3636//testReport/
          Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/3636//artifact/trunk/patchprocess/diffJavacWarnings.txt
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3636//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642188/yarn-1964-docker.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. -1 javac . The applied patch generated 1281 javac compiler warnings (more than the trunk's current 1278 warnings). +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3636//testReport/ Javac warnings: https://builds.apache.org/job/PreCommit-YARN-Build/3636//artifact/trunk/patchprocess/diffJavacWarnings.txt Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3636//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          Removed javac warnings.

          Show
          Abin Shahab added a comment - Removed javac warnings.
          Abin Shahab made changes -
          Attachment yarn-1964-docker.patch [ 12642267 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642267/yarn-1964-docker.patch
          against trunk revision .

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3641//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3641//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642267/yarn-1964-docker.patch against trunk revision . +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/3641//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/3641//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          Does others have comments on it. Arun C Murthy] ?

          Show
          Abin Shahab added a comment - Does others have comments on it. Arun C Murthy ] ?
          Hide
          Dinesh Subhraveti added a comment -

          We are putting together issues across YARN and Docker projects related to this work as a wiki page: https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers. If you happen to be at Dockercon tomorrow, join our Hadoop BoF session for more discussion!

          This blog post has some background as well: https://www.altiscale.com/making-docker-work-yarn/

          Show
          Dinesh Subhraveti added a comment - We are putting together issues across YARN and Docker projects related to this work as a wiki page: https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers . If you happen to be at Dockercon tomorrow, join our Hadoop BoF session for more discussion! This blog post has some background as well: https://www.altiscale.com/making-docker-work-yarn/
          Hide
          haosdent added a comment -

          Cool feature!

          Show
          haosdent added a comment - Cool feature!
          Hide
          haosdent added a comment -

          The patch looks good.

          Show
          haosdent added a comment - The patch looks good.
          Dinesh Subhraveti made changes -
          Link This issue is related to YARN-2140 [ YARN-2140 ]
          Hide
          Allen Wittenauer added a comment -

          I took a look at the patch. After some discussion/clarification from Abin and Dinesh, here are some issues we've identified with the current patch:

          • yarn-site.xml names are not in the yarn.* hierarchy.
          • The environment variables being imported are likely incomplete (HADOOP_PREFIX, at a minimum, should be added to the list).
          • Application Masters cannot currently be run from DockerContainerExecutor. The current patch launches them with the default launcher.
          • There isn't any special distributed cache handling, which could/will expose private data to everyone as well as risk confusing the node manager if the distributed cache contents are changed out from underneath it.
          • Same thing with logging.
          • This hasn't been tested with any level of security (either 'run as the user' or fully kerberized). There are likely more holes/problems that will be discovered after those tests.
          • There is a question as to how to handle importing Java and the Hadoop setup into the container. There is a risk that just adding HADOOP_PREFIX and marking it as RO may break things in certain configurations (specifically logs, tmp, and pids dirs).
          Show
          Allen Wittenauer added a comment - I took a look at the patch. After some discussion/clarification from Abin and Dinesh, here are some issues we've identified with the current patch: yarn-site.xml names are not in the yarn.* hierarchy. The environment variables being imported are likely incomplete (HADOOP_PREFIX, at a minimum, should be added to the list). Application Masters cannot currently be run from DockerContainerExecutor. The current patch launches them with the default launcher. There isn't any special distributed cache handling, which could/will expose private data to everyone as well as risk confusing the node manager if the distributed cache contents are changed out from underneath it. Same thing with logging. This hasn't been tested with any level of security (either 'run as the user' or fully kerberized). There are likely more holes/problems that will be discovered after those tests. There is a question as to how to handle importing Java and the Hadoop setup into the container. There is a risk that just adding HADOOP_PREFIX and marking it as RO may break things in certain configurations (specifically logs, tmp, and pids dirs).
          Abin Shahab made changes -
          Link This issue incorporates YARN-2466 [ YARN-2466 ]
          Abin Shahab made changes -
          Link This issue incorporates YARN-2466 [ YARN-2466 ]
          Abin Shahab made changes -
          Link This issue is part of YARN-2466 [ YARN-2466 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12642267/yarn-1964-docker.patch
          against trunk revision ab638e7.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4759//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4759//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12642267/yarn-1964-docker.patch against trunk revision ab638e7. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The following test timeouts occurred in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestSubmitApplicationWithRMHA +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/4759//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/4759//console This message is automatically generated.
          Hide
          Krisztian Horvath added a comment -

          Will containers be able to communicate with each other, e.g with slider I can run HBase inside containers.

          Show
          Krisztian Horvath added a comment - Will containers be able to communicate with each other, e.g with slider I can run HBase inside containers.
          Hide
          Abin Shahab added a comment -

          Krisztian Horvath, Yes, Containers would be able to talk to one another. They will be launched in net=host mode, and so they will take on the network stack of the host.
          Let me know if you have other questions.

          Show
          Abin Shahab added a comment - Krisztian Horvath , Yes, Containers would be able to talk to one another. They will be launched in net=host mode, and so they will take on the network stack of the host. Let me know if you have other questions.
          Hide
          Abin Shahab added a comment -

          We have decided to create an umbrella issue to cover the integration between YARN and Docker(YARN-2466).
          This task(YARN-1964) has the following scope:
          1) Launch docker containers from YARN with net=host mode. This will allow the container to take on the host's network, and therefore the YARN administrators will not need to set up special networking for docker.
          2) Allow users to provide docker images through the job configuration.
          3) Setup and user guides.

          The rest(secure hadoop, advanced networking) will be handled in other issues under YARN-2466.
          Please add your feedback to this plan on this jira.
          Thanks!
          Abin

          Show
          Abin Shahab added a comment - We have decided to create an umbrella issue to cover the integration between YARN and Docker( YARN-2466 ). This task( YARN-1964 ) has the following scope: 1) Launch docker containers from YARN with net=host mode. This will allow the container to take on the host's network, and therefore the YARN administrators will not need to set up special networking for docker. 2) Allow users to provide docker images through the job configuration. 3) Setup and user guides. The rest(secure hadoop, advanced networking) will be handled in other issues under YARN-2466 . Please add your feedback to this plan on this jira. Thanks! Abin
          Allen Wittenauer made changes -
          Comment [ Pull-request on git: https://github.com/apache/hadoop/pull/2(comments are welcome). ]
          Hide
          Abin Shahab added a comment -

          Patch that scopes down the YARN integration as mentioned above.

          Show
          Abin Shahab added a comment - Patch that scopes down the YARN integration as mentioned above.
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12671475 ]
          Abin Shahab made changes -
          Status Patch Available [ 10002 ] In Progress [ 3 ]
          Hide
          Abin Shahab added a comment -

          patch submitted.

          Show
          Abin Shahab added a comment - patch submitted.
          Abin Shahab made changes -
          Status In Progress [ 3 ] Patch Available [ 10002 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12671475/YARN-1964.patch
          against trunk revision a6049aa.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5147//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5147//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12671475/YARN-1964.patch against trunk revision a6049aa. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5147//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5147//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          Added example and test for JAVA_HOME change for mappers and reducers.

          Show
          Abin Shahab added a comment - Added example and test for JAVA_HOME change for mappers and reducers.
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12671739 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12671739/YARN-1964.patch
          against trunk revision b38e52b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5167//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5167//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12671739/YARN-1964.patch against trunk revision b38e52b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5167//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5167//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          Harmonized changes between yarn-default.xml and YarnConfiguration. Updated docs.

          Show
          Abin Shahab added a comment - Harmonized changes between yarn-default.xml and YarnConfiguration. Updated docs.
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12672275 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12672275/YARN-1964.patch
          against trunk revision 17d1202.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5194//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5194//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12672275/YARN-1964.patch against trunk revision 17d1202. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5194//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5194//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          Patch with all classpath changes inside DCE.

          Show
          Abin Shahab added a comment - Patch with all classpath changes inside DCE.
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12672747 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12672747/YARN-1964.patch
          against trunk revision 2d8e6e2.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 2 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5251//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5251//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12672747/YARN-1964.patch against trunk revision 2d8e6e2. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 2 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5251//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5251//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          This patch simplifies the use-case by exposing only one docker configuration param: The image. Now the user must configure the image completely so that all require resources and environment variables are defined in the image.

          Show
          Abin Shahab added a comment - This patch simplifies the use-case by exposing only one docker configuration param: The image. Now the user must configure the image completely so that all require resources and environment variables are defined in the image.
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12675937 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12675937/YARN-1964.patch
          against trunk revision 8942741.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5466//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5466//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12675937/YARN-1964.patch against trunk revision 8942741. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5466//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5466//console This message is automatically generated.
          Hide
          Sidharta Seethana added a comment -

          Hi Abin,

          I applied the patch to a recent snapshot of 2.6 and tested it out. The patch needs to be re-based (the changes are minor) to fix some minor compilation issues. Also, docker pull for the image mentioned in the example ( altiscale/hadoop-docker ) appears to be large and takes long time to pull - what is the reason for this? Is there a way we can bring down the size of the image?

          thanks,
          -Sid

          Show
          Sidharta Seethana added a comment - Hi Abin, I applied the patch to a recent snapshot of 2.6 and tested it out. The patch needs to be re-based (the changes are minor) to fix some minor compilation issues. Also, docker pull for the image mentioned in the example ( altiscale/hadoop-docker ) appears to be large and takes long time to pull - what is the reason for this? Is there a way we can bring down the size of the image? thanks, -Sid
          Hide
          Abin Shahab added a comment -

          Sid, thanks for looking at it.
          The docker image is sequenceiq/hadoop-docker:2.4.0 with all the HADOOP_* environments exposed. I would not even upload our image if sequenceiq's image exposed all the environment variables.
          So I'm not sure why it takes so much time. I'd say try comparing it with the download time of sequenceiq/hadoop-docker:2.4.0(on a different machine, otherwise you'll get the tag from the docker cache).

          I'll try to rebase the patch soon.

          Show
          Abin Shahab added a comment - Sid, thanks for looking at it. The docker image is sequenceiq/hadoop-docker:2.4.0 with all the HADOOP_* environments exposed. I would not even upload our image if sequenceiq's image exposed all the environment variables. So I'm not sure why it takes so much time. I'd say try comparing it with the download time of sequenceiq/hadoop-docker:2.4.0(on a different machine, otherwise you'll get the tag from the docker cache). I'll try to rebase the patch soon.
          Hide
          Janos Matyas added a comment -

          Hi Abin,

          I have applied the changes to the image the next day Re your request - sorry for being late, Docker.io does not send notifications about comments. Should you need anything in the future please drop me a direct email - janos.matyas@sequenceiq.com - or open a GitHub issue.

          Janos

          Show
          Janos Matyas added a comment - Hi Abin, I have applied the changes to the image the next day Re your request - sorry for being late, Docker.io does not send notifications about comments. Should you need anything in the future please drop me a direct email - janos.matyas@sequenceiq.com - or open a GitHub issue. Janos
          Hide
          Abin Shahab added a comment -

          Patch with merge conflicts fixed. Docs pointing back to sequenceiq/hadoop-docker:2.4.1

          Show
          Abin Shahab added a comment - Patch with merge conflicts fixed. Docs pointing back to sequenceiq/hadoop-docker:2.4.1
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12677620 ]
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12677620/YARN-1964.patch
          against trunk revision 58c0bb9.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          -1 javac. The patch appears to cause the build to fail.

          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5605//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677620/YARN-1964.patch against trunk revision 58c0bb9. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. -1 javac . The patch appears to cause the build to fail. Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5605//console This message is automatically generated.
          Show
          Abin Shahab added a comment - This says it passed: https://builds.apache.org/job/PreCommit-YARN-Build/5605/artifact/patchprocess/trunkJavacWarnings.txt/*view*/ Is there another patch that's making it fail? https://issues.apache.org/jira/browse/HADOOP-10926 ?
          Hide
          Sidharta Seethana added a comment -

          This commit in branch-2.6 : https://github.com/apache/hadoop/commit/29d0164e which changed the signature of an abstract function in ContainerExecutor. It looks like your latest patch fixes this, though. We'll take a look, thanks.

          Show
          Sidharta Seethana added a comment - This commit in branch-2.6 : https://github.com/apache/hadoop/commit/29d0164e which changed the signature of an abstract function in ContainerExecutor. It looks like your latest patch fixes this, though. We'll take a look, thanks.
          Hide
          Sidharta Seethana added a comment -

          Abin,

          There seems to be a typo (a spurious newline) in the patch that causes a build failure :

          • ApplicationConstants.Environment.HADOOP_HDFS_HOME.$()
            + ApplicationConstant
            + s.Environment.HADOOP_HDFS_HOME.$()

          Could you please fix ?

          thanks.

          Show
          Sidharta Seethana added a comment - Abin, There seems to be a typo (a spurious newline) in the patch that causes a build failure : ApplicationConstants.Environment.HADOOP_HDFS_HOME.$() + ApplicationConstant + s.Environment.HADOOP_HDFS_HOME.$() Could you please fix ? thanks.
          Hide
          Abin Shahab added a comment -

          Fixed rebase issue.

          Show
          Abin Shahab added a comment - Fixed rebase issue.
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12677719 ]
          Hide
          Abin Shahab added a comment -

          Fixed.
          I had tested the patch on my box, and that passed. Not sure how it passed
          when there was such an obvious error.

          On Tue, Oct 28, 2014 at 2:39 PM, sidharta seethana (JIRA) <jira@apache.org>

          Show
          Abin Shahab added a comment - Fixed. I had tested the patch on my box, and that passed. Not sure how it passed when there was such an obvious error. On Tue, Oct 28, 2014 at 2:39 PM, sidharta seethana (JIRA) <jira@apache.org>
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12677719/YARN-1964.patch
          against trunk revision 8984e9b.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 1 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          -1 core tests. The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager:

          org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
          org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler
          org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart
          org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5613//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5613//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677719/YARN-1964.patch against trunk revision 8984e9b. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 1 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. -1 core tests . The patch failed these unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestCapacityScheduler org.apache.hadoop.yarn.server.resourcemanager.TestRMRestart org.apache.hadoop.yarn.server.resourcemanager.TestClientRMTokens +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5613//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5613//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          I don't think the broken rm tests are linked to this patch.

          Show
          Abin Shahab added a comment - I don't think the broken rm tests are linked to this patch.
          Hide
          Abin Shahab added a comment -

          Arun C Murthy, Vinod Kumar Vavilapalli, Do you guys have any comments on this?

          Show
          Abin Shahab added a comment - Arun C Murthy , Vinod Kumar Vavilapalli , Do you guys have any comments on this?
          Hide
          Ravi Prakash added a comment -

          Hi Abin! Thanks a lot for your work on this.

          Here are my comments:
          1. Nit: Please document DOCKER_IMAGE_PATTERN . What is that regex for?
          2. Please document toMount()
          3. Remove @VisibleForTesting for public static abstract class ShellScriptBuilder
          4. Please explicitly specify your imports. Not import <package>.*;
          5. If you made writeLaunchEnv static, you wouldn't have to unnecessarily create a DefaultContainerExecutor in TestContainerLaunch
          6. In DockerContainerExecutor:
          a. In init() throw an exception if DOCKER_CONTAINER_EXECUTOR_SCRIPT doesn't exists. That way the NodeManager won't come up.
          b. The comment "// create log dir under app" seems misplaced
          c. You are missing shExec.close() in the finally block (not that it does anything, but still
          d. In writeLaunchEnv, you can remove exclusionSet and call super.writeLaunchEnv
          e. out.close(); could throw an Exception in the finally block.
          f. What's the story on signalContainer(), deleteAsUser() and isContainerProcessAlive() (false is probably the wrong value to return here)? I'm not sure we can commit this patch without those functions.

          Show
          Ravi Prakash added a comment - Hi Abin! Thanks a lot for your work on this. Here are my comments: 1. Nit: Please document DOCKER_IMAGE_PATTERN . What is that regex for? 2. Please document toMount() 3. Remove @VisibleForTesting for public static abstract class ShellScriptBuilder 4. Please explicitly specify your imports. Not import <package>.*; 5. If you made writeLaunchEnv static, you wouldn't have to unnecessarily create a DefaultContainerExecutor in TestContainerLaunch 6. In DockerContainerExecutor: a. In init() throw an exception if DOCKER_CONTAINER_EXECUTOR_SCRIPT doesn't exists. That way the NodeManager won't come up. b. The comment "// create log dir under app" seems misplaced c. You are missing shExec.close() in the finally block (not that it does anything, but still d. In writeLaunchEnv, you can remove exclusionSet and call super.writeLaunchEnv e. out.close(); could throw an Exception in the finally block. f. What's the story on signalContainer(), deleteAsUser() and isContainerProcessAlive() (false is probably the wrong value to return here)? I'm not sure we can commit this patch without those functions.
          Hide
          Chun Chen added a comment -

          Hi, Abin Shahab, thanks for the patch. I’m also working on this and we are running dockers on Yarn now. Based on our experience, here are my comments:
          1. Since yarn.nodemanager.docker-container-executor.image-name is a application specified container launch environment argument which would be exported in launch_container.sh and bash doesn't allow dot separated environment variables, so you might need to change it to ApplicationConstants.Environment.DOCKER_IMAGE_NAME.
          2. Remove “--net=host” from docker run command cause cluster administrator might not want docker containers to use host network directly.
          3. Define a new environment maybe ApplicationConstants.Environment.DOCKER_RUN_ARGS to allow application to specify customer options of docker run command such as -P, -e, etc. And in this way, we can also specify "—net=host” from application side.
          4. Remove localDirMount from docker run command, there is no need to mount local dir.
          5. Use ApplicationConstants.Environment.HADOOP_YARN_HOME instead of “HADOOP_YARN_HOME"

              exclusionSet.add("HADOOP_YARN_HOME");
              exclusionSet.add("HADOOP_COMMON_HOME");
              exclusionSet.add("HADOOP_HDFS_HOME");
              exclusionSet.add("HADOOP_COMMON_HOME");
              exclusionSet.add("JAVA_HOME");
          
          Show
          Chun Chen added a comment - Hi, Abin Shahab , thanks for the patch. I’m also working on this and we are running dockers on Yarn now. Based on our experience, here are my comments: 1. Since yarn.nodemanager.docker-container-executor.image-name is a application specified container launch environment argument which would be exported in launch_container.sh and bash doesn't allow dot separated environment variables, so you might need to change it to ApplicationConstants.Environment.DOCKER_IMAGE_NAME. 2. Remove “--net=host” from docker run command cause cluster administrator might not want docker containers to use host network directly. 3. Define a new environment maybe ApplicationConstants.Environment.DOCKER_RUN_ARGS to allow application to specify customer options of docker run command such as -P, -e, etc. And in this way, we can also specify "—net=host” from application side. 4. Remove localDirMount from docker run command, there is no need to mount local dir. 5. Use ApplicationConstants.Environment.HADOOP_YARN_HOME instead of “HADOOP_YARN_HOME" exclusionSet.add( "HADOOP_YARN_HOME" ); exclusionSet.add( "HADOOP_COMMON_HOME" ); exclusionSet.add( "HADOOP_HDFS_HOME" ); exclusionSet.add( "HADOOP_COMMON_HOME" ); exclusionSet.add( "JAVA_HOME" );
          Hide
          Abin Shahab added a comment -

          Chun Chen Thanks a lot for your feedback.
          I have made some of the changes you’ve requested.
          1) I am excluding this if you look at writeLaunchEnv
          2) In this first phase, we want to make this patch as simple as possible, and so we don’t want to add the burden of additional networking on the admin, which is why we are not changing this.
          3) Similarly, we do not want to allow an unlimited number of configuration options to the docker container. We want to do this in future. Feel free to create a new task under the umbrella jira, and submit your pull request.
          4) I have found it useful for certain jobs. Please look at my test.
          5) Agreed, and changed.

          [~ravihoo], I have made most of the changes you’ve requested. Let me know if anything else is needed.

          Show
          Abin Shahab added a comment - Chun Chen Thanks a lot for your feedback. I have made some of the changes you’ve requested. 1) I am excluding this if you look at writeLaunchEnv 2) In this first phase, we want to make this patch as simple as possible, and so we don’t want to add the burden of additional networking on the admin, which is why we are not changing this. 3) Similarly, we do not want to allow an unlimited number of configuration options to the docker container. We want to do this in future. Feel free to create a new task under the umbrella jira, and submit your pull request. 4) I have found it useful for certain jobs. Please look at my test. 5) Agreed, and changed. [~ravihoo] , I have made most of the changes you’ve requested. Let me know if anything else is needed.
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12680305 ]
          Hide
          Abin Shahab added a comment -

          I meant, Ravi Prakash, I have made most of the changes you’ve requested. Let me know if anything else is needed.

          Show
          Abin Shahab added a comment - I meant, Ravi Prakash , I have made most of the changes you’ve requested. Let me know if anything else is needed.
          Hide
          Hadoop QA added a comment -

          -1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12680305/YARN-1964.patch
          against trunk revision c3d4750.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          -1 findbugs. The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5797//testReport/
          Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/5797//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5797//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - -1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12680305/YARN-1964.patch against trunk revision c3d4750. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. -1 findbugs . The patch appears to introduce 1 new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5797//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-YARN-Build/5797//artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5797//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          Added nullcheck

          Show
          Abin Shahab added a comment - Added nullcheck
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12680359 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12680359/YARN-1964.patch
          against trunk revision df36edf.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5803//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5803//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12680359/YARN-1964.patch against trunk revision df36edf. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5803//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5803//console This message is automatically generated.
          Hide
          Abin Shahab added a comment -

          fixed imports.

          Show
          Abin Shahab added a comment - fixed imports.
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12680823 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12680823/YARN-1964.patch
          against trunk revision 58e9bf4.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5815//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5815//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12680823/YARN-1964.patch against trunk revision 58e9bf4. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5815//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5815//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          Thanks Abin! The patch is looking really good now. However the documentation doesn't seem to be compiling for me. Once that is figured out, I'm a +1. I am looking to commit it EOD today to trunk, branch-2, branch-2.6. I'd like to commit it to 2.6 also and request a respin of the RC.

          Show
          Ravi Prakash added a comment - Thanks Abin! The patch is looking really good now. However the documentation doesn't seem to be compiling for me. Once that is figured out, I'm a +1. I am looking to commit it EOD today to trunk, branch-2, branch-2.6. I'd like to commit it to 2.6 also and request a respin of the RC.
          Hide
          Karthik Kambatla added a comment -

          Ravi Prakash - I would prefer backporting it to branch-2.6 only if it goes into 2.6.0 release, so we can avoid including features in point releases. In any case, the plan is to release 2.7.0 soon after 2.6.0.

          Show
          Karthik Kambatla added a comment - Ravi Prakash - I would prefer backporting it to branch-2.6 only if it goes into 2.6.0 release, so we can avoid including features in point releases. In any case, the plan is to release 2.7.0 soon after 2.6.0.
          Hide
          Ravi Prakash added a comment -

          Hi Karthik! That's fair. I'll ask Arun if he is willing to re-spin 2.6.0.

          Show
          Ravi Prakash added a comment - Hi Karthik! That's fair. I'll ask Arun if he is willing to re-spin 2.6.0.
          Hide
          Abin Shahab added a comment -

          Added docs to the site.xml

          Show
          Abin Shahab added a comment - Added docs to the site.xml
          Abin Shahab made changes -
          Attachment YARN-1964.patch [ 12680880 ]
          Hide
          Hadoop QA added a comment -

          +1 overall. Here are the results of testing the latest attachment
          http://issues.apache.org/jira/secure/attachment/12680880/YARN-1964.patch
          against trunk revision 99d9d0c.

          +1 @author. The patch does not contain any @author tags.

          +1 tests included. The patch appears to include 3 new or modified test files.

          +1 javac. The applied patch does not increase the total number of javac compiler warnings.

          +1 javadoc. There were no new javadoc warning messages.

          +1 eclipse:eclipse. The patch built with eclipse:eclipse.

          +1 findbugs. The patch does not introduce any new Findbugs (version 2.0.3) warnings.

          +1 release audit. The applied patch does not increase the total number of release audit warnings.

          +1 core tests. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.

          +1 contrib tests. The patch passed contrib unit tests.

          Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5817//testReport/
          Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5817//console

          This message is automatically generated.

          Show
          Hadoop QA added a comment - +1 overall . Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12680880/YARN-1964.patch against trunk revision 99d9d0c. +1 @author . The patch does not contain any @author tags. +1 tests included . The patch appears to include 3 new or modified test files. +1 javac . The applied patch does not increase the total number of javac compiler warnings. +1 javadoc . There were no new javadoc warning messages. +1 eclipse:eclipse . The patch built with eclipse:eclipse. +1 findbugs . The patch does not introduce any new Findbugs (version 2.0.3) warnings. +1 release audit . The applied patch does not increase the total number of release audit warnings. +1 core tests . The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager. +1 contrib tests . The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-YARN-Build/5817//testReport/ Console output: https://builds.apache.org/job/PreCommit-YARN-Build/5817//console This message is automatically generated.
          Hide
          Ravi Prakash added a comment -

          I'm a +1 on this patch. I'll commit it to trunk and branch-2 soon. Soon as I get confirmation from Arun, I'll commit it into branch-2.6 as well.

          Show
          Ravi Prakash added a comment - I'm a +1 on this patch. I'll commit it to trunk and branch-2 soon. Soon as I get confirmation from Arun, I'll commit it into branch-2.6 as well.
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-trunk-Commit #6517 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6517/)
          YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed)

          • hadoop-project/src/site/site.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-trunk-Commit #6517 (See https://builds.apache.org/job/Hadoop-trunk-Commit/6517/ ) YARN-1964 . Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed) hadoop-project/src/site/site.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          Hide
          Ravi Prakash added a comment -

          I've committed this to trunk and branch-2. I wasn't sure about whether to put the release notes under release 2.6 or 2.7, but on a leap of faith, I've put it under 2.6 right now. I'll fix if it Arun declines to respin an RC.

          Show
          Ravi Prakash added a comment - I've committed this to trunk and branch-2. I wasn't sure about whether to put the release notes under release 2.6 or 2.7, but on a leap of faith, I've put it under 2.6 right now. I'll fix if it Arun declines to respin an RC.
          Ravi Prakash made changes -
          Assignee Abin Shahab [ ashahab ] Ravi Prakash [ raviprak ]
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #3 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/3/)
          YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-project/src/site/site.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #3 (See https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/3/ ) YARN-1964 . Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-project/src/site/site.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Yarn-trunk #741 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/741/)
          YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-project/src/site/site.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/CHANGES.txt
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Yarn-trunk #741 (See https://builds.apache.org/job/Hadoop-Yarn-trunk/741/ ) YARN-1964 . Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-project/src/site/site.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/CHANGES.txt
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #3 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/3/)
          YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-project/src/site/site.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #3 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/3/ ) YARN-1964 . Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-project/src/site/site.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Hdfs-trunk #1931 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1931/)
          YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-project/src/site/site.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Hdfs-trunk #1931 (See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1931/ ) YARN-1964 . Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-project/src/site/site.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          Hide
          Hudson added a comment -

          FAILURE: Integrated in Hadoop-Mapreduce-trunk #1955 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1955/)
          YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed)

          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-project/src/site/site.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Show
          Hudson added a comment - FAILURE: Integrated in Hadoop-Mapreduce-trunk #1955 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1955/ ) YARN-1964 . Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-project/src/site/site.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          Hide
          Hudson added a comment -

          SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #3 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/3/)
          YARN-1964. Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed)

          • hadoop-project/src/site/site.xml
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
          • hadoop-yarn-project/CHANGES.txt
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java
          • hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          Show
          Hudson added a comment - SUCCESS: Integrated in Hadoop-Mapreduce-trunk-Java8 #3 (See https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/3/ ) YARN-1964 . Create Docker analog of the LinuxContainerExecutor in YARN (raviprak: rev 53f64ee516d03f6ec87b41d77c214aa2fe4fa0ed) hadoop-project/src/site/site.xml hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutorWithMocks.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/DockerContainerExecutor.apt.vm hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/ContainerLaunch.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/DockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/TestDockerContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml hadoop-yarn-project/CHANGES.txt hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/main/java/org/apache/hadoop/yarn/server/nodemanager/ContainerExecutor.java hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager/src/test/java/org/apache/hadoop/yarn/server/nodemanager/containermanager/launcher/TestContainerLaunch.java
          Hide
          Ravi Prakash added a comment -

          Since Arun agreed to spin an rc1, I have committed it to branch-2.6 as well. Closing as fixed. Thanks a lot Abin, Jay, Arun, Vinod, Wangda, Junping, Allen, Dinesh, haosdent, Tsuyoshi, Krisztian, Sidharta, Janos, Chun, Karthik and everybody else I might have missed.

          Show
          Ravi Prakash added a comment - Since Arun agreed to spin an rc1, I have committed it to branch-2.6 as well. Closing as fixed. Thanks a lot Abin, Jay, Arun, Vinod, Wangda, Junping, Allen, Dinesh, haosdent, Tsuyoshi, Krisztian, Sidharta, Janos, Chun, Karthik and everybody else I might have missed.
          Ravi Prakash made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Fix Version/s 2.6.0 [ 12327197 ]
          Resolution Fixed [ 1 ]
          Hide
          Ravi Prakash added a comment -

          Also committed to branch-2.6.0

          Show
          Ravi Prakash added a comment - Also committed to branch-2.6.0
          Arun C Murthy made changes -
          Assignee Ravi Prakash [ raviprak ] Abin Shahab [ ashahab ]
          Hide
          Arun C Murthy added a comment -

          Ravi Prakash I'm concerned this is coming in VERY late into 2.6... we've been in closedown mode for a while. The only mitigating factor is that this is fairly isolated since it's a new ContainerExecutor, and we can label it as an alpha feature. Any other objections to pulling this into 2.6?

          Show
          Arun C Murthy added a comment - Ravi Prakash I'm concerned this is coming in VERY late into 2.6... we've been in closedown mode for a while. The only mitigating factor is that this is fairly isolated since it's a new ContainerExecutor , and we can label it as an alpha feature. Any other objections to pulling this into 2.6?
          Hide
          Vinod Kumar Vavilapalli added a comment -

          Arun C Murthy Ravi Prakash Agreed. Granted Abin Shahab and Sidharta Seethana have been at it for a while, but it is late to get it in.

          I did a quick scan anyways, the patch is not risky to any of the existing code-base, so I am reluctantly okay leaving this in 2.6. Let's avoid doing this in future though.

          One thing of note - we should explicitly call this and a few other features as explicit alpha level features for setting right expectation for the users and the compatibility story.

          Show
          Vinod Kumar Vavilapalli added a comment - Arun C Murthy Ravi Prakash Agreed. Granted Abin Shahab and Sidharta Seethana have been at it for a while, but it is late to get it in. I did a quick scan anyways, the patch is not risky to any of the existing code-base, so I am reluctantly okay leaving this in 2.6. Let's avoid doing this in future though. One thing of note - we should explicitly call this and a few other features as explicit alpha level features for setting right expectation for the users and the compatibility story.
          Arun C Murthy made changes -
          Status Resolved [ 5 ] Closed [ 6 ]
          Hide
          Chen He added a comment -

          Thank Abin Shahab for the patch. It will be more clear if we add "only support single node cluster" in the [working example]. Since the "fs.defaultFS" is configured to be "hdfs://localhost:9000" in sequenceiq/hadoop-docker:2.6.0. If we use it in a distributed environment, the MRAppMaster will not find hdfs and fail.

          Show
          Chen He added a comment - Thank Abin Shahab for the patch. It will be more clear if we add "only support single node cluster" in the [working example] . Since the "fs.defaultFS" is configured to be "hdfs://localhost:9000" in sequenceiq/hadoop-docker:2.6.0. If we use it in a distributed environment, the MRAppMaster will not find hdfs and fail.
          Hide
          Abin Shahab added a comment -

          Chen He
          We at altiscale are running this on a 6 node cluster all the time.
          The working example uses this image: sequenceiq/hadoop-docker:2.4.1
          That absolutely works in a multi-node cluster.

          I don't know anything about sequenceiq/hadoop-docker:2.6.0.

          Abin

          Show
          Abin Shahab added a comment - Chen He We at altiscale are running this on a 6 node cluster all the time. The working example uses this image: sequenceiq/hadoop-docker:2.4.1 That absolutely works in a multi-node cluster. I don't know anything about sequenceiq/hadoop-docker:2.6.0. Abin
          Hide
          Janos Matyas added a comment -

          Hi Chen He / [~ashabab]

          There should be no fundamental difference - it's only a newer Hadoop codebase.

          Janos

          PS: Thanks for the patch

          Show
          Janos Matyas added a comment - Hi Chen He / [~ashabab] There should be no fundamental difference - it's only a newer Hadoop codebase. Janos PS: Thanks for the patch
          Hide
          Abin Shahab added a comment -

          Hey Janos Matyas, if you have a chance, can you give the patch a try(hadoop-2.6) with your newest images? I really liked the images you guys released, so it'd be good if it's all compatible.
          Thanks!

          Show
          Abin Shahab added a comment - Hey Janos Matyas , if you have a chance, can you give the patch a try(hadoop-2.6) with your newest images? I really liked the images you guys released, so it'd be good if it's all compatible. Thanks!
          Hide
          Janos Matyas added a comment -

          Hi Abin Shahab, yes will give it a try and let you know how it works.

          Show
          Janos Matyas added a comment - Hi Abin Shahab , yes will give it a try and let you know how it works.
          Hide
          Attila Kanto added a comment -

          Hi Abin Shahab,

          I have tested it with the configuration described in: http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html and it works with sequenceiq/hadoop-docker:2.6.0 as well.

          Same teragen test was executed what is described in documentation:
          ~ $ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar teragen -Dmapreduce.map.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.6.0" -Dyarn.app.mapreduce.am.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.6.0" 1 /tmp/out

          The execution was successful and also the containers are launched correctly by DCE:
          ~ $ docker ps -a
          CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
          20ce5a4b1654 sequenceiq/hadoop-docker:2.6.0 "bash /tmp/hadoop-ro 3 seconds ago Up 2 seconds container_1418483101027_0003_01_000003
          82ec1e3eec73 sequenceiq/hadoop-docker:2.6.0 "bash /tmp/hadoop-ro 3 seconds ago Up 3 seconds container_1418483101027_0003_01_000002
          f7c436fd5086 sequenceiq/hadoop-docker:2.6.0 "bash /tmp/hadoop-ro 10 seconds ago Up 9 seconds container_1418483101027_0003_01_000001
          b8516f47cb86 my_dce:latest "/etc/bootstrap.sh - 3 minutes ago Up 3 minutes jolly_bell

          Attila

          Show
          Attila Kanto added a comment - Hi Abin Shahab , I have tested it with the configuration described in: http://hadoop.apache.org/docs/stable/hadoop-yarn/hadoop-yarn-site/DockerContainerExecutor.html and it works with sequenceiq/hadoop-docker:2.6.0 as well. Same teragen test was executed what is described in documentation: ~ $ hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.0.jar teragen -Dmapreduce.map.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.6.0" -Dyarn.app.mapreduce.am.env="yarn.nodemanager.docker-container-executor.image-name=sequenceiq/hadoop-docker:2.6.0" 1 /tmp/out The execution was successful and also the containers are launched correctly by DCE: ~ $ docker ps -a CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 20ce5a4b1654 sequenceiq/hadoop-docker:2.6.0 "bash /tmp/hadoop-ro 3 seconds ago Up 2 seconds container_1418483101027_0003_01_000003 82ec1e3eec73 sequenceiq/hadoop-docker:2.6.0 "bash /tmp/hadoop-ro 3 seconds ago Up 3 seconds container_1418483101027_0003_01_000002 f7c436fd5086 sequenceiq/hadoop-docker:2.6.0 "bash /tmp/hadoop-ro 10 seconds ago Up 9 seconds container_1418483101027_0003_01_000001 b8516f47cb86 my_dce:latest "/etc/bootstrap.sh - 3 minutes ago Up 3 minutes jolly_bell Attila
          Hide
          Abin Shahab added a comment -

          Thanks for testing!

          On Sat, Dec 13, 2014 at 7:20 AM, Attila Kanto (JIRA) <jira@apache.org>

          Show
          Abin Shahab added a comment - Thanks for testing! On Sat, Dec 13, 2014 at 7:20 AM, Attila Kanto (JIRA) <jira@apache.org>
          Hide
          Abin Shahab added a comment -

          Hi Chen He, Did this fix your issue?
          Abin

          Show
          Abin Shahab added a comment - Hi Chen He , Did this fix your issue? Abin
          Hide
          Chen He added a comment -

          Not really. I will let you know what is the reason once I figure it out. Thank you for asking, Abin Shahab.

          Show
          Chen He added a comment - Not really. I will let you know what is the reason once I figure it out. Thank you for asking, Abin Shahab .
          Beckham007 made changes -
          Link This issue relates to YARN-3080 [ YARN-3080 ]
          Ravi Prakash made changes -
          Assignee Abin Shahab [ ashahab ] Ravi Prakash [ raviprak ]
          Ravi Prakash made changes -
          Assignee Ravi Prakash [ raviprak ] Abin Shahab [ ashahab ]
          Vinod Kumar Vavilapalli made changes -
          Link This issue relates to YARN-1983 [ YARN-1983 ]
          Transition Time In Source Status Execution Times Last Executer Last Execution Date
          Open Open In Progress In Progress
          5d 3h 45m 1 Abin Shahab 24/Apr/14 19:41
          Patch Available Patch Available Open Open
          2m 33s 1 Abin Shahab 24/Apr/14 19:47
          Open Open Patch Available Patch Available
          14m 3s 1 Abin Shahab 24/Apr/14 20:01
          Patch Available Patch Available In Progress In Progress
          154d 22h 7m 1 Abin Shahab 26/Sep/14 18:08
          In Progress In Progress Patch Available Patch Available
          3m 56s 2 Abin Shahab 26/Sep/14 18:09
          Patch Available Patch Available Resolved Resolved
          47d 53m 1 Ravi Prakash 12/Nov/14 18:02
          Resolved Resolved Closed Closed
          18d 9h 8m 1 Arun C Murthy 01/Dec/14 03:11

            People

            • Assignee:
              Abin Shahab
              Reporter:
              Arun C Murthy
            • Votes:
              11 Vote for this issue
              Watchers:
              98 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development