Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3611

Support Docker Containers In LinuxContainerExecutor

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: yarn
    • Labels:

      Description

      Support Docker Containers In LinuxContainerExecutor

      LinuxContainerExecutor provides useful functionality today with respect to localization, cgroups based resource management and isolation for CPU, network, disk etc. as well as security with a well-defined mechanism to execute privileged operations using the container-executor utility. Bringing docker support to LinuxContainerExecutor lets us use all of this functionality when running docker containers under YARN, while not requiring users and admins to configure and use a different ContainerExecutor.

      There are several aspects here that need to be worked through :

      • Mechanism(s) to let clients request docker-specific functionality - we could initially implement this via environment variables without impacting the client API.
      • Security - both docker daemon as well as application
      • Docker image localization
      • Running a docker container via container-executor as a specified user
      • “Isolate” the docker container in terms of CPU/network/disk/etc
      • Communicating with and/or signaling the running container (ensure correct pid handling)
      • Figure out workarounds for certain performance-sensitive scenarios like HDFS short-circuit reads
      • All of these need to be achieved without changing the current behavior of LinuxContainerExecutor

        Attachments

          Issue Links

          1.
          Add support for container runtimes in YARN Sub-task Resolved Sidharta Seethana
          2.
          Add docker container support to container-executor Sub-task Resolved Abin Shahab
          3.
          Add docker container runtime support to LinuxContainterExecutor Sub-task Resolved Sidharta Seethana
          4.
          container-executor should print output of docker logs if the docker container exits with non-0 exit status Sub-task Resolved Varun Vasudev
          5.
          Add support for different network setups when launching the docker container Sub-task Resolved Sidharta Seethana
          6.
          Add support for controlling capabilities for docker containers Sub-task Resolved Sidharta Seethana
          7.
          container-executor does not clean up docker operation command files. Sub-task Resolved Sidharta Seethana
          8.
          Allow whitelisted users to run privileged docker containers. Sub-task Resolved Sidharta Seethana
          9.
          Allow users to enter containers as UID:GID pair instead of by username Sub-task Resolved luhuichun
          10.
          Add additional logging to container launch implementations in container-executor Sub-task Resolved Sidharta Seethana
          11.
          Add cgroups support for docker containers Sub-task Resolved Sidharta Seethana
          12.
          Directories that are mounted in docker containers need to be more restrictive/container-specific Sub-task Resolved Sidharta Seethana
          13.
          Add support for configurable read-only mounts when launching Docker containers Sub-task Resolved Billie Rinaldi
          14.
          Container recovery is broken with delegating container runtime Sub-task Resolved Sidharta Seethana
          15.
          Fix signal handling for docker containers Sub-task Resolved Shane Kumpf
          16.
          Mount /sys/fs/cgroup into Docker containers as read only mount Sub-task Resolved luhuichun
          17.
          Document Use of Docker with LinuxContainerExecutor Sub-task Resolved Daniel Templeton
          18.
          Mount usercache and NM filecache directories into Docker container Sub-task Resolved Sidharta Seethana
          19.
          Log Docker run command when container fails Sub-task Resolved Varun Vasudev
          20.
          Decouple host user and Docker container user Sub-task Resolved Zhankun Tang
          21.
          Improve handling of the Docker container life cycle Sub-task Resolved Shane Kumpf
          22.
          Remove bind-mount /etc/passwd for Docker containers Sub-task Resolved Zhankun Tang
          23.
          Allow for specifying the docker client configuration directory Sub-task Resolved Shane Kumpf
          24.
          Add support for docker inspect command Sub-task Resolved Shane Kumpf
          25.
          Add support for docker rm Sub-task Resolved Shane Kumpf
          26.
          Rename DockerStopCommandTest to TestDockerStopCommand Sub-task Resolved Shane Kumpf
          27.
          Allow user provided Docker volume mount list Sub-task Resolved Shane Kumpf
          28.
          Fix failing unit test in TestDockerContainerRuntime Sub-task Resolved Sidharta Seethana
          29.
          Remove package line length checkstyle rule Sub-task Resolved Shane Kumpf
          30.
          Support a white list of admin approved mounts Sub-task Resolved Shane Kumpf
          31.
          Add support for Docker pull Sub-task Resolved luhuichun
          32.
          Add support for Docker image clean up Sub-task Resolved Unassigned
          33.
          Support the Docker Live Restore feature Sub-task Resolved Shane Kumpf
          34.
          Improve signaling of short lived containers Sub-task Resolved Shane Kumpf
          35.
          Improve test coverage and add utility classes for common Docker operations Sub-task Resolved Shane Kumpf
          36.
          Improve Diagonstic by moving Error stack trace from NM to slider AM Sub-task Resolved Eric Yang
          37.
          add mounting of HDFS Short-Circuit path for docker containers Sub-task Resolved Jaeboo Jeong
          38.
          Add default volume mount list Sub-task Resolved Eric Badger
          39.
          Add support to turn off launching privileged containers in the container-executor Sub-task Resolved Varun Vasudev
          40.
          container-executor fails for docker when command length > 4096 B Sub-task Resolved Eric Badger
          41.
          Make CGROUPS_ROOT_DIRECTORY configurable in DockerLinuxContainerRuntime Sub-task Resolved Eric Badger
          42.
          Add ability to specify volumes to mount for DockerContainerRuntime Sub-task Resolved Eric Yang
          43.
          Container-executor doesn't remove Docker containers that error out early Sub-task Resolved Eric Badger
          44.
          Add support for a volume blacklist for docker containers Sub-task Resolved Unassigned
          45.
          Make Docker target directory for cgroups configurable by yarn-site.xml Sub-task Resolved Shane Kumpf
          46.
          Consider /sys/fs/cgroup as the default CGroup mount path Sub-task Resolved Shane Kumpf
          47.
          Make the cgroup mount into Docker containers configurable Sub-task Resolved Shane Kumpf
          48.
          Add support for docker to have no capabilities Sub-task Resolved Eric Badger
          49.
          Docker permitted volumes don't properly check for directories Sub-task Resolved Eric Badger
          50.
          NM fails to successfully kill tasks that run over their memory limit Sub-task Resolved Unassigned
          51.
          Fix issues with docker commands executed by container-executor Sub-task Resolved Shane Kumpf
          52.
          Trim configuration values in DockerLinuxContainerRuntime Sub-task Resolved Tianyin Xu
          53.
          Docker container privileged mode and --user flag contradict each other Sub-task Resolved Eric Yang
          54.
          Enable user re-mapping for Docker containers by default Sub-task Resolved Eric Yang
          55.
          Security check for trusted docker image Sub-task Resolved Eric Yang
          56.
          NM gets backed up deleting docker containers Sub-task Open Eric Badger
          57.
          Support ENTRY_POINT for docker container Sub-task Resolved Eric Yang
          58.
          Docker Stop grace period should be configurable Sub-task Resolved Eric Badger
          59.
          Avoid using docker volume --format option to run against older docker releases Sub-task Resolved Wangda Tan
          60.
          Add support for setting the PID namespace mode Sub-task Resolved Billie Rinaldi
          61.
          Enable user re-mapping for Docker containers in yarn-default.xml Sub-task Resolved Eric Yang
          62.
          Docker host network can not obtain IP address for RegistryDNS Sub-task Resolved Eric Yang
          63.
          TestDockerContainerRuntime test failures due to UID lookup of a non-existent user Sub-task Resolved Shane Kumpf
          64.
          Remove automatic mounting of the cgroups root directory into Docker containers Sub-task Resolved Shane Kumpf
          65.
          Make the YARN mounts added to Docker containers more restrictive Sub-task Resolved Shane Kumpf
          66.
          Fix exit code handling for short lived Docker containers Sub-task Resolved Shane Kumpf
          67.
          Trusted image log message repeated multiple times Sub-task Resolved Shane Kumpf
          68.
          Remove call to docker logs on failure in container-executor Sub-task Resolved Shane Kumpf
          69.
          Fix failing test TestDockerContainerRuntime#testLaunchContainerWithDockerTokens Sub-task Resolved Shane Kumpf
          70.
          Add no-new-privileges flag to docker run Sub-task Resolved Eric Badger
          71.
          Support ContainerRelaunch for Docker containers Sub-task Resolved Shane Kumpf
          72.
          Docker container name(--name) needs to be DNS friendly for DNS resolution to work in user defined networks. Sub-task Resolved Suma Shivaprasad
          73.
          YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS should not use commas as separators Sub-task Resolved Jim Brennan
          74.
          Setting hostname of docker container breaks for --net=host in docker 1.13 Sub-task Resolved Jim Brennan
          75.
          Docker launch fails when user private filecache directory is missing Sub-task Resolved Jason Lowe
          76.
          Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount Sub-task Resolved Zian Chen
          77.
          Docker ".cmd" files should not be put in hadoop.tmp.dir Sub-task Resolved Eric Badger
          78.
          Add support for Docker env-file switch Sub-task Resolved Unassigned
          79.
          Sending a kill does not immediately kill docker containers Sub-task Resolved Eric Badger
          80.
          Docker container launch use popen have risk of shell expansion Sub-task Resolved Eric Yang
          81.
          Docker does not support hostnames greater than 64 characters Sub-task Resolved Shane Kumpf
          82.
          MRAppMaster fails when using UID:GID pair within docker container Sub-task Resolved Unassigned
          83.
          Revisit liveliness checks for Docker containers Sub-task Resolved Shane Kumpf
          84.
          Docker container launch fails due to .cmd file creation failure Sub-task Resolved Jason Lowe
          85.
          Remove unused environment variables from the Docker runtime Sub-task Resolved Eric Badger
          86.
          get_docker_command refactoring Sub-task Resolved Eric Badger
          87.
          Docker client configuration can still be set incorrectly Sub-task Resolved Shane Kumpf
          88.
          Privileged docker containers' jobSubmitDir does not get successfully cleaned up Sub-task Resolved Unassigned
          89.
          YARN should have ability to run images only from a whitelist docker registries Sub-task Resolved Unassigned
          90.
          Using docker image from a non-privileged registry, the launch_command is not honored Sub-task Resolved Eric Yang
          91.
          stdout.txt, stderr.txt logs of a launched docker container is coming with primary group of submit user instead of hadoop Sub-task Resolved Eric Yang
          92.
          Dshell docker container gets marked as lost after NM restart Sub-task Resolved Shane Kumpf
          93.
          Priviledged container app launch is failing intermittently Sub-task Resolved Eric Yang

            Activity

              People

              • Assignee:
                sidharta-s Sidharta Seethana
                Reporter:
                sidharta-s Sidharta Seethana
              • Votes:
                5 Vote for this issue
                Watchers:
                92 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: