[YARN-4725] [Umbrella] Auto-­restart of containers

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Unassigned
• Reporter: Vinod Kumar Vavilapalli
• Created: 23/Feb/16 22:20
• Updated: 18/Apr/16 01:22

Description

See overview doc at YARN-4692, copying the sub-section to track all related efforts.

Today, when a container (process­-tree) dies, NodeManager assumes that the container’s allocation is also expired, and reports accordingly to the ResourceManager which then releases the allocation. For service containers, this is undesirable in many cases. Long running containers may exit for various reasons, crash and need to restart but forcing them to go through the complete scheduling cycle, resource localization etc is both unnecessary and expensive. (​Task) ​For services it will be good to have NodeManagers automatically restart containers. This looks a lot like inittab / daemon­tools at the system level.

We will need to enable app­-specific policies (very similar to the handling of AM restarts at YARN level) for restarting containers automatically but limit such restarts if a container dies too often in a short interval of time.

YARN-3998 is an existing ticket that looks at some if not all of this functionality.

Issue Links

• is part of: YARN-4692 [Umbrella] Simplified and first-class support for services in YARN

Sub-Tasks

• Add support in the NodeManager to re-launch containers: Resolved
• Auto-restart of containers should work across NM restarts.: Resolved
• Support sliding window retry capability for container restart: Resolved
• Add support for a minimum retry interval for container retries: Resolved
• Relook at containerLaunchStartTime in the NM: Open
• Allow services to release/kill specific containers: Open
• Use the common sliding window retry policy for AM restart: Open
• Inform AM of container relaunch: Open