[YARN-1198] Capacity Scheduler headroom calculation does not work as expected - ASF JIRA

XML

Word

Printable

JSON

Today headroom calculation (for the app) takes place only when

However there are potentially lot of situations which are not considered for this calculation

If a container finishes then headroom for that application will change and should be notified to the AM accordingly.
If a single user has submitted multiple applications (app1 and app2) to the same queue then
- If app1's container finishes then not only app1's but also app2's AM should be notified about the change in headroom.
- Similarly if a container is assigned to any applications app1/app2 then both AM should be notified about their headroom.
- To simplify the whole communication process it is ideal to keep headroom per User per LeafQueue so that everyone gets the same picture (apps belonging to same user and submitted in same queue).
If a new user submits an application to the queue then all applications submitted by all users in that queue should be notified of the headroom change.
Also today headroom is an absolute number ( I think it should be normalized but then this is going to be not backward compatible..)
Also when admin user refreshes queue headroom has to be updated.

These all are the potential bugs in headroom calculations

1.	CapacityScheduler headroom doesn't account for other AM's running	Closed	Chen He
2.	availableResources sent to applicationMaster in heartbeat should exclude blacklistedNodes free memory.	Open	Unassigned
3.	CapacityScheduler may report incorrect queueMaxCap if there is hierarchy queue structure	Closed	Craig Welch
4.	Recalculate headroom more frequently to keep it accurate	Closed	Craig Welch
5.	CapacityScheduler deadlock when computing absolute max avail capacity (fix for trunk/branch-2)	Closed	Wangda Tan
6.	Queue's current-max-limit should be updated before allocate reserved container.	Open	Wangda Tan
7.	Queue's current-max-limit should be updated when container allocated/released in another queue	Open	Wangda Tan
8.	AvailableResource of QueueMetrics should consider queue's current-max-limit	Open	Unassigned