[YARN-1503] Support making additional 'LocalResources' available to running containers

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Jian He
• Reporter: Siddharth Seth
• Created: 12/Dec/13 21:20
• Updated: 24/May/17 03:09

Description

We have a use case, where additional resources (jars, libraries etc) need to be made available to an already running container. Ideally, we'd like this to be done via YARN (instead of having potentially multiple containers per node download resources on their own).

Proposal:
NM to support an additional API where a list of resources can be specified. Something like "localiceResource(ContainerId, Map<String, LocalResource>)
NM would also require an additional API to get state for these resources - "getLocalizationState(ContainerId)" - which returns the current state of all local resources for the specified container(s).

Issue Links

• duplicates: YARN-5532 Make ResourceLocalizationService inside NodeManager independent of containers
• relates to: YARN-6630 Container worker dir could not recover when NM restart
• relates to: YARN-6624 The implementation of getLocalizationStatus
• relates to: YARN-6606 The implementation of LocalizationStatus in ContainerStatusProto

Sub-Tasks

• Add localize API to the ContainerManagementProtocol: Resolved
• Allow resource localization while container is running: Resolved
• Support LinuxContainerExecutor to create symlinks for continuously localized resources: Patch Available