[YARN-8472] YARN Container Phase 2

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.3.0

Release Note

- Improved debugging Docker container on YARN
- Improved security for running Docker containers
- Improved cgroup management for docker container.

• Assignee: Eric Yang
• Reporter: Eric Yang
• Created: 28/Jun/18 17:43
• Updated: 22/Jan/20 21:53
• Resolved: 22/Jan/20 21:53

Description

In YARN-3611, we have implemented basic Docker container support for YARN.  This story is the next phase to improve container usability.

Several area for improvements are:

1. Software defined network support
2. Interactive shell to container
3. User management sss/nscd integration
4. Runc/containerd support
5. Metrics/Logs integration with Timeline service v2
6. Docker container profiles
7. Docker cgroup management

Issue Links

• relates to: YARN-3611 Support Docker Containers In LinuxContainerExecutor

Sub-Tasks

• check docker container's exit code when writing to cgroup task files: Resolved
• Allow administrators to control available container runtimes and set defaults for all containers: Resolved
• Force removal of docker containers that do not get removed on first try: Resolved
• Update documentation and yarn-default related to the Docker runtime: Resolved
• DockerClient still touches hadoop.tmp.dir: Resolved
• Separate white list for docker.trusted.registries and docker.privileged-container.registries: Resolved
• Privileged, trusted containers should be supported only in ENTRYPOINT mode: Resolved
• Add port mapping handling when docker container use bridge network: Resolved
• Document best practice for user management: Resolved
• Interactive docker shell: Resolved
• Create an interface to provide cluster information to application: Resolved
• Automate docker network configuration through YARN API: Resolved
• Update Docker examples to use image which exists: Resolved
• Cleanup ENTRYPOINT documentation: Resolved
• Add support for tmpfs mounts with the Docker runtime: Resolved
• Docker image life cycle management on HDFS: Resolved
• Expose container's hostname to applications running within the docker container: Resolved
• Add support for network-alias in docker run for user defined networks: Resolved
• NM gets backed up deleting docker containers: Resolved
• DelayedProcessKiller is executed for Docker containers even though docker stop sends a KILL signal after the specified grace period: Resolved
• Improve the error message when a bind mount is not whitelisted: Resolved
• Improve the documentation for the docker.trusted.registries configuration: Resolved
• When STOPSIGNAL is not present then docker inspect returns an extra line feed: Resolved
• In some cases docker kill is used to stop non-privileged containers instead of sending the signal directly: Resolved
• javadoc comments in docker-util.h is confusing: Resolved
• Automatically convert the launch command to the exec form when using entrypoint support: Resolved
• Add a flag to use local docker image instead of getting latest from registry: Resolved
• Support multi-homed network for docker container: Resolved
• publish all exposed ports to random ports when using bridge network: Resolved
• Docker container rm command should be executed after stop: Resolved
• Carriage Return character in launch command cause node manager to become unhealthy: Resolved
• Container reinit always fails in branch-3.2 and trunk: Resolved
• Docker run doesn't pull down latest image if the image exists locally: Resolved
• Implement logic to keep docker image consistent in application that uses :latest tag: Resolved
• Disable PATH variable to be passed to Docker container: Resolved
• Docker container exited with failure does not get clean up correctly: Resolved
• Reinit of container stuck in localization: Resolved
• Enhance documentation of Docker on YARN support: Resolved
• Remove reapDockerContainerNoPid left behind by YARN-9074: Resolved