[SLIDER-1168] Avoid or eliminate expensive YARN get all applications call

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: Slider 0.91
• Assignee: Unassigned
• Reporter: Gour Saha
• Created: 24/Sep/16 01:47
• Updated: 10/Nov/16 19:40

Description

YARN get all applications call will prove to be very costly in a cluster with hundreds or thousands of Slider apps running. We need to come up with a strategy to avoid or eliminate these expensive calls. This would mean that the application id needs to be stored in Slider's state store in HDFS (or some other better strategy). This is specifically more critical when Slider Client calls are made from a service like the YARN native REST API service (YARN-4793).