[YARN-2466] Umbrella issue for Yarn launched Docker Containers

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.4.1
• Labels: Docker
• Assignee: Unassigned
• Reporter: Abin Shahab
• Created: 29/Aug/14 00:18
• Updated: 16/May/18 19:58
• Resolved: 04/Apr/18 12:35

Description

Docker (https://www.docker.io/) is, increasingly, a very popular container technology.

In context of YARN, the support for Docker will provide a very elegant solution to allow applications to package their software into a Docker container (entire Linux file system incl. custom versions of perl, python etc.) and use it as a blueprint to launch all their YARN containers with requisite software environment. This provides both consistency (all YARN containers will have the same software environment) and isolation (no interference with whatever is installed on the physical machine).

In addition to software isolation mentioned above, Docker containers will provide resource, network, and user-namespace isolation.

Docker provides resource isolation through cgroups, similar to LinuxContainerExecutor. This prevents one job from taking other jobs resource(memory and CPU) on the same hadoop cluster.

User-namespace isolation will ensure that the root on the container is mapped an unprivileged user on the host. This is currently being added to Docker.

Network isolation will ensure that one user’s network traffic is completely isolated from another user’s network traffic.

Last but not the least, the interaction of Docker and Kerberos will have to be worked out. These Docker containers must work in a secure hadoop environment.

Additional details are here: https://wiki.apache.org/hadoop/dineshs/IsolatingYarnAppsInDockerContainers

Issue Links

• incorporates: YARN-2478 Nested containers should be supported
• incorporates: YARN-2479 DockerContainerExecutor must support handling of distributed cache
• incorporates: YARN-2482 DockerContainerExecutor configuration
• incorporates: YARN-2981 DockerContainerExecutor must support a Cluster-wide default Docker image
• incorporates: YARN-2477 DockerContainerExecutor must support secure mode
• incorporates: YARN-2480 DockerContainerExecutor must support user namespaces
• incorporates: YARN-2481 YARN should allow defining the location of java
• incorporates: YARN-2718 Create a CompositeConatainerExecutor that combines DockerContainerExecutor and DefaultContainerExecutor
• incorporates: YARN-3095 Enable DockerContainerExecutor to update Docker image
• incorporates: YARN-1964 Create Docker analog of the LinuxContainerExecutor in YARN
• incorporates: YARN-2878 Fix DockerContainerExecutor.apt.vm formatting
• is part of: YARN-3290 DockerContainerExecutor should optionally limit memory and cpu
• is part of: YARN-3291 DockerContainerExecutor should run as a non-root user inside the container
• is related to: YARN-3611 Support Docker Containers In LinuxContainerExecutor

Sub-Tasks

• Document and fix indentation in the DockerContainerExecutor code: Resolved
• Docker images should be downloaded during localization: Resolved
• TestDockerContainerExecutor should run automatically if it can detect docker in the usual place: Resolved
• TestDockerContainerExecutor should clean test docker image from local repository after test is done: Resolved
• Nested containers should be supported: Resolved
• DockerContainerExecutor must support handling of distributed cache: Resolved
• DockerContainerExecutor configuration: Resolved
• DockerContainerExecutor should allow user specify "docker run" parameters: Resolved