Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3611

Support Docker Containers In LinuxContainerExecutor

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: yarn
    • Labels:
      None

      Description

      Support Docker Containers In LinuxContainerExecutor

      LinuxContainerExecutor provides useful functionality today with respect to localization, cgroups based resource management and isolation for CPU, network, disk etc. as well as security with a well-defined mechanism to execute privileged operations using the container-executor utility. Bringing docker support to LinuxContainerExecutor lets us use all of this functionality when running docker containers under YARN, while not requiring users and admins to configure and use a different ContainerExecutor.

      There are several aspects here that need to be worked through :

      • Mechanism(s) to let clients request docker-specific functionality - we could initially implement this via environment variables without impacting the client API.
      • Security - both docker daemon as well as application
      • Docker image localization
      • Running a docker container via container-executor as a specified user
      • “Isolate” the docker container in terms of CPU/network/disk/etc
      • Communicating with and/or signaling the running container (ensure correct pid handling)
      • Figure out workarounds for certain performance-sensitive scenarios like HDFS short-circuit reads
      • All of these need to be achieved without changing the current behavior of LinuxContainerExecutor

        Issue Links

        1.
        Add support for container runtimes in YARN Sub-task Resolved Sidharta Seethana
         
        2.
        Add docker container support to container-executor Sub-task Resolved Abin Shahab
         
        3.
        Add docker container runtime support to LinuxContainterExecutor Sub-task Resolved Sidharta Seethana
         
        4. Add localization support for docker images Sub-task Open Shane Kumpf
         
        5.
        container-executor should print output of docker logs if the docker container exits with non-0 exit status Sub-task Resolved Varun Vasudev
         
        6.
        Add support for different network setups when launching the docker container Sub-task Resolved Sidharta Seethana
         
        7.
        Add support for controlling capabilities for docker containers Sub-task Resolved Sidharta Seethana
         
        8.
        container-executor does not clean up docker operation command files. Sub-task Resolved Sidharta Seethana
         
        9.
        Allow whitelisted users to run privileged docker containers. Sub-task Resolved Sidharta Seethana
         
        10.
        Allow users to enter containers as UID:GID pair instead of by username Sub-task Resolved luhuichun
         
        11.
        Add additional logging to container launch implementations in container-executor Sub-task Resolved Sidharta Seethana
         
        12.
        Add cgroups support for docker containers Sub-task Resolved Sidharta Seethana
         
        13.
        Directories that are mounted in docker containers need to be more restrictive/container-specific Sub-task Resolved Sidharta Seethana
         
        14.
        Add support for configurable read-only mounts when launching Docker containers Sub-task Resolved Billie Rinaldi
         
        15.
        Container recovery is broken with delegating container runtime Sub-task Resolved Sidharta Seethana
         
        16.
        Fix signal handling for docker containers Sub-task Resolved Shane Kumpf
         
        17.
        Mount /sys/fs/cgroup into Docker containers as read only mount Sub-task Resolved luhuichun
         
        18. Add port mapping handling when docker container use bridge network Sub-task Open Unassigned
         
        19.
        Document Use of Docker with LinuxContainerExecutor Sub-task Resolved Daniel Templeton
         
        20.
        Mount usercache and NM filecache directories into Docker container Sub-task Resolved Sidharta Seethana
         
        21.
        Log Docker run command when container fails Sub-task Resolved Varun Vasudev
         
        22.
        Decouple host user and Docker container user Sub-task Resolved Zhankun Tang
         
        23. Improve handling of the Docker container life cycle Sub-task Open Shane Kumpf
         
        24.
        Remove bind-mount /etc/passwd for Docker containers Sub-task Resolved Zhankun Tang
         
        25. Allow for specifying the docker client configuration directory Sub-task Patch Available Shane Kumpf
         
        26.
        Add support for docker inspect command Sub-task Resolved Shane Kumpf
         
        27.
        Add support for docker rm Sub-task Resolved Shane Kumpf
         
        28.
        Rename DockerStopCommandTest to TestDockerStopCommand Sub-task Resolved Shane Kumpf
         
        29.
        Allow user provided Docker volume mount list Sub-task Resolved Shane Kumpf
         
        30.
        Fix failing unit test in TestDockerContainerRuntime Sub-task Resolved Sidharta Seethana
         
        31.
        Remove package line length checkstyle rule Sub-task Resolved Shane Kumpf
         
        32.
        Support a white list of admin approved mounts Sub-task Resolved Shane Kumpf
         
        33.
        Add support for Docker pull Sub-task Resolved luhuichun
         
        34. Add support for Docker image clean up Sub-task Open Shane Kumpf
         
        35.
        Add support for Docker image clean up Sub-task Resolved Unassigned
         
        36. Support the Docker Live Restore feature Sub-task Open Shane Kumpf
         
        37. Improve signaling of short lived containers Sub-task Open Shane Kumpf
         
        38.
        Improve test coverage and add utility classes for common Docker operations Sub-task Resolved Shane Kumpf
         
        39. Improve Diagonstic by moving Error stack trace from NM to slider AM Sub-task Open Unassigned
         
        40.
        add mounting of HDFS Short-Circuit path for docker containers Sub-task Resolved Jaeboo Jeong
         
        41. check docker container's exit code when writing to cgroup task files Sub-task Patch Available Jaeboo Jeong
         
        42. Add default volume mount list Sub-task Open Eric Badger
         
        43.
        Add support to turn off launching privileged containers in the container-executor Sub-task Resolved Varun Vasudev
         
        44.
        container-executor fails for docker when command length > 4096 B Sub-task Resolved Eric Badger
         
        45.
        Make CGROUPS_ROOT_DIRECTORY configurable in DockerLinuxContainerRuntime Sub-task Resolved Eric Badger
         
        46.
        Add ability to specify volumes to mount for DockerContainerRuntime Sub-task Resolved Eric Yang
         
        47. Container-executor doesn't remove Docker containers that error out early Sub-task Open Eric Badger
         
        48. Add support for a volume blacklist for docker containers Sub-task Patch Available Unassigned
         
        49. Make Docker target directory for cgroups configurable by yarn-site.xml Sub-task Open Unassigned
         
        50. Consider /sys/fs/cgroup as the default CGroup mount path Sub-task Open Unassigned
         
        51. Make the cgroup mount into Docker containers configurable Sub-task Open Unassigned
         
        52.
        Add support for docker to have no capabilities Sub-task Resolved Eric Badger
         
        53.
        Docker permitted volumes don't properly check for directories Sub-task Resolved Eric Badger
         
        54.
        NM fails to successfully kill tasks that run over their memory limit Sub-task Resolved Unassigned
         
        55.
        Fix issues with docker commands executed by container-executor Sub-task Resolved Shane Kumpf
         
        56.
        Trim configuration values in DockerLinuxContainerRuntime Sub-task Resolved Tianyin Xu
         
        57. Docker container privileged mode and --user flag contradict each other Sub-task Patch Available Eric Yang
         
        58.
        Enable user re-mapping for Docker containers by default Sub-task Resolved Eric Yang
         
        59. Security check for untrusted docker image Sub-task Open Unassigned
         

          Activity

          Hide
          sidharta-s Sidharta Seethana added a comment -

          /cc Abin Shahab, Vinod Kumar Vavilapalli : Please chime in

          Show
          sidharta-s Sidharta Seethana added a comment - /cc Abin Shahab , Vinod Kumar Vavilapalli : Please chime in
          Hide
          aw Allen Wittenauer added a comment -

          I'm pretty much against the idea. It's much easier to change DCE to add these features rather than muck with LCE, which has been stable and consistent for years. In fact, YARN-3291 is pretty much doing exactly that.

          So I'm inclined to close this as a dupe of that JIRA.

          Show
          aw Allen Wittenauer added a comment - I'm pretty much against the idea. It's much easier to change DCE to add these features rather than muck with LCE, which has been stable and consistent for years. In fact, YARN-3291 is pretty much doing exactly that. So I'm inclined to close this as a dupe of that JIRA.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Hi Allen Wittenauer ,

          I agree with you that we should ensure that current LinuxContainerExecutor functionality isn’t broken - but at the same time, I believe it is important to add useful new features that bring more value to users. This has already been happening to LinuxContainerExecutor
          and related functionality - refactored resource handler/cgroups handler, support for new resources types.

          About YARN-3291 : The patch for this JIRA already moves in the direction described in this JIRA. DockerContainerExecutor is changed to be a child of LinuxContainerExecutor (along with some minor changes to LinuxContainerExecutor itself). In addition, there are changes to the native code in the linux-specific container-executor tool which has so far only been used by LinuxContainerExecutor. I don’t believe this JIRA is a dupe of YARN-3291 - the scope differs quite a bit - more sub-tasks are to be added to this JIRA for various pieces of functionality that need to be built.

          thanks,
          -Sidharta

          Show
          sidharta-s Sidharta Seethana added a comment - Hi Allen Wittenauer , I agree with you that we should ensure that current LinuxContainerExecutor functionality isn’t broken - but at the same time, I believe it is important to add useful new features that bring more value to users. This has already been happening to LinuxContainerExecutor and related functionality - refactored resource handler/cgroups handler, support for new resources types. About YARN-3291 : The patch for this JIRA already moves in the direction described in this JIRA. DockerContainerExecutor is changed to be a child of LinuxContainerExecutor (along with some minor changes to LinuxContainerExecutor itself). In addition, there are changes to the native code in the linux-specific container-executor tool which has so far only been used by LinuxContainerExecutor. I don’t believe this JIRA is a dupe of YARN-3291 - the scope differs quite a bit - more sub-tasks are to be added to this JIRA for various pieces of functionality that need to be built. thanks, -Sidharta
          Hide
          vinodkv Vinod Kumar Vavilapalli added a comment -

          Just looked at this and YARN-3291 JIRA.

          The main direction of both of these JIRAs as I understand is to reuse code from (a) the container-executor binary and (b) the java LinuxContainerExecutor code; so as to setup things correctly. This was one of my comments in the original docker integration JIRAs itself.

          May be it is just the JIRA title that is throwing us off a bit. Both JIRAs are advocating similar idea though. Can we establish a clear relationship between the two? Sidharta Seethana and Abin Shahab.

          Show
          vinodkv Vinod Kumar Vavilapalli added a comment - Just looked at this and YARN-3291 JIRA. The main direction of both of these JIRAs as I understand is to reuse code from (a) the container-executor binary and (b) the java LinuxContainerExecutor code; so as to setup things correctly. This was one of my comments in the original docker integration JIRAs itself. May be it is just the JIRA title that is throwing us off a bit. Both JIRAs are advocating similar idea though. Can we establish a clear relationship between the two? Sidharta Seethana and Abin Shahab .
          Hide
          ashahab Abin Shahab added a comment -

          I agree with Vinod Kumar Vavilapalli. Both Jiras have the same objective, and I am fine with either.
          I like the proposal in this Jira, because it makes Docker a first class citizen of Hadoop. However, I'm fine with keeping it in DefaultContainerExecutor also if merging this into LCE slows the whole process down.
          Also, network and disk resource management is new in LCE, so we should not make those a blocker for Docker integration.
          My main concern across all Docker issues is that are we making incremental progress. We have alpha, and we should make incremental progress towards beta.

          Show
          ashahab Abin Shahab added a comment - I agree with Vinod Kumar Vavilapalli . Both Jiras have the same objective, and I am fine with either. I like the proposal in this Jira, because it makes Docker a first class citizen of Hadoop. However, I'm fine with keeping it in DefaultContainerExecutor also if merging this into LCE slows the whole process down. Also, network and disk resource management is new in LCE, so we should not make those a blocker for Docker integration. My main concern across all Docker issues is that are we making incremental progress. We have alpha, and we should make incremental progress towards beta.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Abin Shahab and I have been working together on this for the past few weeks. (We demoed this recently as well). I am going to file sub tasks so that we can make progress.

          thanks,
          -Sidharta

          Show
          sidharta-s Sidharta Seethana added a comment - Abin Shahab and I have been working together on this for the past few weeks. (We demoed this recently as well). I am going to file sub tasks so that we can make progress. thanks, -Sidharta
          Hide
          sidharta-s Sidharta Seethana added a comment -

          Looks like I forgot to link to the slides/demo from hadoop summit. Here it is, in case somebody is interested in taking a look : https://prezi.com/2mxvb0n_q1rt/yarn-and-the-docker-ecosystem/

          Show
          sidharta-s Sidharta Seethana added a comment - Looks like I forgot to link to the slides/demo from hadoop summit. Here it is, in case somebody is interested in taking a look : https://prezi.com/2mxvb0n_q1rt/yarn-and-the-docker-ecosystem/
          Hide
          templedf Daniel Templeton added a comment - - edited

          Has the LinuxContainerExecutor Docker support been documented yet, or should I file a JIRA to add docs? I didn't see anything.

          Show
          templedf Daniel Templeton added a comment - - edited Has the LinuxContainerExecutor Docker support been documented yet, or should I file a JIRA to add docs? I didn't see anything.
          Hide
          sidharta-s Sidharta Seethana added a comment -

          hi Daniel Templeton, please go ahead and file a JIRA for adding docs. Thanks!

          Show
          sidharta-s Sidharta Seethana added a comment - hi Daniel Templeton , please go ahead and file a JIRA for adding docs. Thanks!
          Hide
          aw Allen Wittenauer added a comment -

          Why is this not being done in a branch?

          Show
          aw Allen Wittenauer added a comment - Why is this not being done in a branch?
          Hide
          hrsharma Hitesh Sharma added a comment -

          Hi folks,

          Docker is now available on Windows and is fully supported by Docker INC (I'm talking about launching Windows containers via Docker).

          https://www.docker.com/microsoft

          Unfortunately in the current design Docker is being limited to Linux only. I think we need to revisit this and have a way to share the same code across Docker support for Windows and Linux. Another goal to keep in mind is to have DockerContainerExecutor be completely OS agnostic. As in certain cases Docker client might actually be talking to a daemon on a remote machine or a VM (which maybe Linux or Windows). Would love to hear some thoughts on how to achieve Docker support for Windows by reusing all the good work being done here.

          Thanks!

          Show
          hrsharma Hitesh Sharma added a comment - Hi folks, Docker is now available on Windows and is fully supported by Docker INC (I'm talking about launching Windows containers via Docker). https://www.docker.com/microsoft Unfortunately in the current design Docker is being limited to Linux only. I think we need to revisit this and have a way to share the same code across Docker support for Windows and Linux. Another goal to keep in mind is to have DockerContainerExecutor be completely OS agnostic. As in certain cases Docker client might actually be talking to a daemon on a remote machine or a VM (which maybe Linux or Windows). Would love to hear some thoughts on how to achieve Docker support for Windows by reusing all the good work being done here. Thanks!

            People

            • Assignee:
              sidharta-s Sidharta Seethana
              Reporter:
              sidharta-s Sidharta Seethana
            • Votes:
              2 Vote for this issue
              Watchers:
              70 Start watching this issue

              Dates

              • Created:
                Updated:

                Development