[YARN-5501] Container Pooling in YARN

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Assignee: Hitesh Sharma
• Reporter: Arun Suresh
• Created: 09/Aug/16 23:46
• Updated: 23/Feb/17 03:19

Description

This JIRA proposes a method for reducing the container launch latency in YARN. It introduces a notion of pooling Unattached Pre-Initialized Containers.

Proposal in brief:

• Have a Pre-Initialized Container Factory service within the NM to create these unattached containers.
• The NM would then advertise these containers as special resource types (this should be possible via YARN-3926).
• When a start container request is received by the node manager for launching a container requesting this specific type of resource, it will take one of these unattached pre-initialized containers from the pool, and use it to service the container request.
• Once the request is complete, the pre-initialized container would be released and ready to serve another request.

This capability would help reduce container launch latencies and thereby allow for development of more interactive applications on YARN.