[YARN-9834] Allow using a pool of local users to run Yarn Secure Container in secure mode

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: nodemanager
• Labels: pull-request-available
• Assignee: shanyu zhao
• Reporter: shanyu zhao
• Created: 14/Sep/19 02:20
• Updated: 27/Jan/22 19:38

Description

Yarn Secure Container allows separation of different user's local files and container processes running on the same node manager. This depends on an out of band service such as SSSD/Winbind to sync all domain users to local machine that runs Yarn node manager. Hadoop code only works with local users.

Winbind/SSSD user sync has lots of overhead, especially for large corporations. Also if running Yarn node manager inside Kubernetes cluster (meaning node managers running inside Docker container), it doesn't make sense for each Docker container to domain join with Active Directory and sync a whole copy of domain users to the Docker container.

We need an optional light-weighted approach to enable Yarn Secure Container in secure mode, as an alternative to AD domain join and SSSD/Winbind based user-sync service.

Today, class LinuxContainerExecutor already supports running Yarn container process as one designated local user in non-secure mode.
We can add new configurations to Yarn, such that with LinuxContainerExecutor we can pre-create a pool of local users on each Yarn node manager. At runtime, Yarn node manager allocates a local user to run the container process, for the domain user that submits the application. When all containers of that user are finished and all files belonging to that user are deleted, we can release the allocation and allow other users to use the same local user to run their Yarn containers.

Please look at attached design doc for more details.

 

Issue Links

• links to: GitHub Pull Request #1446