[MYRIAD-12] Implement auto scaling

• Type: Improvement
• Status: Open
• Resolution: Unresolved
• Assignee: Santosh Marella
• Reporter: Santosh Marella
• Created: 21/Oct/14 00:06
• Updated: 13/Apr/18 10:14

Description

• Pluggable criteria to detect "YARN starvation".. some ideas:
• if all NMs are full and more than X number of resource requests waiting in the queue
• if all NMs are full and more than Y number of apps are waiting in the queue
• How to scale? i.e. "how many" NM instances are required to bring down the starvation
• "detect the extent of starvation" by querying the resource requests in the pipieline and scale up accordingly