[YARN-3611] Support Docker Containers In LinuxContainerExecutor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: yarn
• Labels: Docker
• Assignee: Sidharta Seethana
• Reporter: Sidharta Seethana
• Created: 09/May/15 00:26
• Updated: 20/Aug/18 22:56
• Resolved: 11/Jul/18 21:56

Description

Support Docker Containers In LinuxContainerExecutor

LinuxContainerExecutor provides useful functionality today with respect to localization, cgroups based resource management and isolation for CPU, network, disk etc. as well as security with a well-defined mechanism to execute privileged operations using the container-executor utility. Bringing docker support to LinuxContainerExecutor lets us use all of this functionality when running docker containers under YARN, while not requiring users and admins to configure and use a different ContainerExecutor.

There are several aspects here that need to be worked through :

• Mechanism(s) to let clients request docker-specific functionality - we could initially implement this via environment variables without impacting the client API.
• Security - both docker daemon as well as application
• Docker image localization
• Running a docker container via container-executor as a specified user
• “Isolate” the docker container in terms of CPU/network/disk/etc
• Communicating with and/or signaling the running container (ensure correct pid handling)
• Figure out workarounds for certain performance-sensitive scenarios like HDFS short-circuit reads
• All of these need to be achieved without changing the current behavior of LinuxContainerExecutor

Issue Links

• blocks: SPARK-20277 Allow Spark on YARN to be launched with Docker
• duplicates: YARN-5209 Transmission ContainerExecutor Class Parameters By The Client
• incorporates: YARN-7221 Add security check for privileged docker container
• incorporates: YARN-7446 Docker container privileged mode and --user flag contradict each other
• incorporates: YARN-7516 Security check for trusted docker image
• is duplicated by: YARN-3201 add args for DistributedShell to specify a image for tasks that will run on docker
• is related to: YARN-7430 Enable user re-mapping for Docker containers by default
• is related to: AMBARI-17353 First class support for YARN hosted services
• is related to: YARN-7677 Docker image cannot set HADOOP_CONF_DIR
• is related to: YARN-8472 YARN Container Phase 2
• relates to: YARN-2466 Umbrella issue for Yarn launched Docker Containers
• relates to: YARN-3291 DockerContainerExecutor should run as a non-root user inside the container

Sub-Tasks

• Add support for container runtimes in YARN: Resolved
• Add docker container support to container-executor: Resolved
• Add docker container runtime support to LinuxContainterExecutor: Resolved
• container-executor should print output of docker logs if the docker container exits with non-0 exit status: Resolved
• Add support for different network setups when launching the docker container: Resolved
• Add support for controlling capabilities for docker containers: Resolved
• container-executor does not clean up docker operation command files.: Resolved
• Allow whitelisted users to run privileged docker containers.: Resolved
• Allow users to enter containers as UID:GID pair instead of by username: Resolved
• Add additional logging to container launch implementations in container-executor: Resolved
• Add cgroups support for docker containers: Resolved
• Directories that are mounted in docker containers need to be more restrictive/container-specific: Resolved
• Add support for configurable read-only mounts when launching Docker containers: Resolved
• Container recovery is broken with delegating container runtime: Resolved
• Fix signal handling for docker containers: Resolved
• Mount /sys/fs/cgroup into Docker containers as read only mount: Resolved
• Document Use of Docker with LinuxContainerExecutor: Resolved
• Mount usercache and NM filecache directories into Docker container: Resolved
• Log Docker run command when container fails: Resolved
• Decouple host user and Docker container user: Resolved
• Improve handling of the Docker container life cycle: Resolved
• Remove bind-mount /etc/passwd for Docker containers: Resolved
• Allow for specifying the docker client configuration directory: Resolved
• Add support for docker inspect command: Resolved
• Add support for docker rm: Resolved
• Rename DockerStopCommandTest to TestDockerStopCommand: Resolved
• Allow user provided Docker volume mount list: Resolved
• Fix failing unit test in TestDockerContainerRuntime: Resolved
• Remove package line length checkstyle rule: Resolved
• Support a white list of admin approved mounts: Resolved
• Add support for Docker pull: Resolved
• Add support for Docker image clean up: Resolved
• Support the Docker Live Restore feature: Resolved
• Improve signaling of short lived containers: Resolved
• Improve test coverage and add utility classes for common Docker operations: Resolved
• Improve Diagonstic by moving Error stack trace from NM to slider AM: Resolved
• add mounting of HDFS Short-Circuit path for docker containers: Resolved
• Add default volume mount list: Resolved
• Add support to turn off launching privileged containers in the container-executor: Resolved
• container-executor fails for docker when command length > 4096 B: Resolved
• Make CGROUPS_ROOT_DIRECTORY configurable in DockerLinuxContainerRuntime: Resolved
• Add ability to specify volumes to mount for DockerContainerRuntime: Resolved
• Container-executor doesn't remove Docker containers that error out early: Resolved
• Add support for a volume blacklist for docker containers: Resolved
• Make Docker target directory for cgroups configurable by yarn-site.xml: Resolved
• Consider /sys/fs/cgroup as the default CGroup mount path: Resolved
• Make the cgroup mount into Docker containers configurable: Resolved
• Add support for docker to have no capabilities: Resolved
• Docker permitted volumes don't properly check for directories: Resolved
• NM fails to successfully kill tasks that run over their memory limit: Resolved
• Fix issues with docker commands executed by container-executor: Resolved
• Trim configuration values in DockerLinuxContainerRuntime: Resolved
• Docker container privileged mode and --user flag contradict each other: Resolved
• Enable user re-mapping for Docker containers by default: Resolved
• Security check for trusted docker image: Resolved
• Support ENTRY_POINT for docker container: Resolved
• Docker Stop grace period should be configurable: Resolved
• Avoid using docker volume --format option to run against older docker releases: Resolved
• Add support for setting the PID namespace mode: Resolved
• Enable user re-mapping for Docker containers in yarn-default.xml: Resolved
• Docker host network can not obtain IP address for RegistryDNS: Resolved
• TestDockerContainerRuntime test failures due to UID lookup of a non-existent user: Resolved
• Remove automatic mounting of the cgroups root directory into Docker containers: Resolved
• Make the YARN mounts added to Docker containers more restrictive: Resolved
• Fix exit code handling for short lived Docker containers: Resolved
• Trusted image log message repeated multiple times: Resolved
• Remove call to docker logs on failure in container-executor: Resolved
• Fix failing test TestDockerContainerRuntime#testLaunchContainerWithDockerTokens: Resolved
• Add no-new-privileges flag to docker run: Resolved
• Support ContainerRelaunch for Docker containers: Resolved
• Docker container name(--name) needs to be DNS friendly for DNS resolution to work in user defined networks.: Resolved
• YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS should not use commas as separators: Resolved
• Setting hostname of docker container breaks for --net=host in docker 1.13: Resolved
• Docker launch fails when user private filecache directory is missing: Resolved
• Allow regular expression matching in container-executor.cfg for devices and named docker volumes mount: Resolved
• Docker ".cmd" files should not be put in hadoop.tmp.dir: Resolved
• Add support for Docker env-file switch: Resolved
• Sending a kill does not immediately kill docker containers: Resolved
• Docker container launch use popen have risk of shell expansion: Resolved
• Docker does not support hostnames greater than 64 characters: Resolved
• MRAppMaster fails when using UID:GID pair within docker container: Resolved
• Revisit liveliness checks for Docker containers: Resolved
• Docker container launch fails due to .cmd file creation failure: Resolved
• Remove unused environment variables from the Docker runtime: Resolved
• get_docker_command refactoring: Resolved
• Docker client configuration can still be set incorrectly: Resolved
• Privileged docker containers' jobSubmitDir does not get successfully cleaned up: Resolved
• YARN should have ability to run images only from a whitelist docker registries: Resolved
• Using docker image from a non-privileged registry, the launch_command is not honored: Resolved
• stdout.txt, stderr.txt logs of a launched docker container is coming with primary group of submit user instead of hadoop: Resolved
• Dshell docker container gets marked as lost after NM restart: Resolved
• Priviledged container app launch is failing intermittently: Resolved