[YARN-3611] Support Docker Containers In LinuxContainerExecutor

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: yarn
• Labels: Docker
• Assignee: Sidharta Seethana
• Reporter: Sidharta Seethana
• Created: 09/May/15 00:26
• Updated: 20/Aug/18 22:56
• Resolved: 11/Jul/18 21:56

Description

Support Docker Containers In LinuxContainerExecutor

LinuxContainerExecutor provides useful functionality today with respect to localization, cgroups based resource management and isolation for CPU, network, disk etc. as well as security with a well-defined mechanism to execute privileged operations using the container-executor utility. Bringing docker support to LinuxContainerExecutor lets us use all of this functionality when running docker containers under YARN, while not requiring users and admins to configure and use a different ContainerExecutor.

There are several aspects here that need to be worked through :

• Mechanism(s) to let clients request docker-specific functionality - we could initially implement this via environment variables without impacting the client API.
• Security - both docker daemon as well as application
• Docker image localization
• Running a docker container via container-executor as a specified user
• “Isolate” the docker container in terms of CPU/network/disk/etc
• Communicating with and/or signaling the running container (ensure correct pid handling)
• Figure out workarounds for certain performance-sensitive scenarios like HDFS short-circuit reads
• All of these need to be achieved without changing the current behavior of LinuxContainerExecutor

Issue Links

• blocks: SPARK-20277 Allow Spark on YARN to be launched with Docker
• duplicates: YARN-5209 Transmission ContainerExecutor Class Parameters By The Client
• incorporates: YARN-7221 Add security check for privileged docker container
• incorporates: YARN-7446 Docker container privileged mode and --user flag contradict each other
• incorporates: YARN-7516 Security check for trusted docker image
• is duplicated by: YARN-3201 add args for DistributedShell to specify a image for tasks that will run on docker
• is related to: YARN-7430 Enable user re-mapping for Docker containers by default
• is related to: AMBARI-17353 First class support for YARN hosted services
• is related to: YARN-7677 Docker image cannot set HADOOP_CONF_DIR
• is related to: YARN-8472 YARN Container Phase 2
• relates to: YARN-2466 Umbrella issue for Yarn launched Docker Containers
• relates to: YARN-3291 DockerContainerExecutor should run as a non-root user inside the container