[YARN-7506] Overhaul the design of the Linux container-executor regarding Docker and future runtimes

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: nodemanager
• Labels: Docker container-executor
• Assignee: Unassigned
• Reporter: Miklos Szegedi
• Created: 15/Nov/17 23:10
• Updated: 20/Nov/17 19:47

Description

I raise this topic to discuss a potential improvement of the container executor tool in node manager.
container-executor has two main purposes. It executes Linux system calls not available from Java, and it executes tasks available to root that are not available to the yarn user. Historically container-executor did both by doing impersonation. The yarn user is separated from root because it runs network services, so the yarn user should be restricted by design. Because of this it has it's own config file container-executor.cfg writable by root only that specifies what actions are allowed for the yarn user. However, the requirements have changed with Docker and that raises the following questions:

1. The Docker feature of YARN requires root permissions to access the Docker socket but it does not run any system calls, so could the Docker related code in container-executor be refactored into a separate Java process ran as root? Java would make the development much faster and more secure.

2. The Docker feature only needs the Docker unix socket. It is not a good idea to let the yarn user directly access the socket, since that would elevate its privileges to root. However, the Java tool running as root mentioned in the previous question could act as a proxy on the Docker socket operating directly on the Docker REST API eliminating the need to use the Docker CLI.

Issue Links

• relates to: YARN-5673 [Umbrella] Re-write container-executor to improve security, extensibility, and portability