[MAPREDUCE-1105] CapacityScheduler: It should be possible to set queue hard-limit beyond it's actual capacity

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.21.0
• Fix Version/s: 0.21.0
• Component/s: capacity-sched
• Hadoop Flags: Reviewed

Release Note

Replaced the existing max task limits variables "mapred.capacity-scheduler.queue.<queue-name>.max.map.slots" and "mapred.capacity-scheduler.queue.<queue-name>.max.reduce.slots" with "mapred.capacity-scheduler.queue.<queue-name>.maximum-capacity" .

max task limit variables were used to throttle the queue, i.e, these were the hard limit and not allowing queue to grow further.
maximum-capacity variable defines a limit beyond which a queue cannot use the capacity of the cluster. This provides a means to limit how much excess capacity a queue can use.
 
maximum-capacity variable behavior is different from max task limit variables, as maximum-capacity is a percentage and it grows and shrinks in absolute terms based on total cluster capacity.Also same maximum-capacity percentage is applied to both map and reduce.

• Assignee: rahul k singh
• Reporter: Arun Murthy
• Created: 14/Oct/09 00:03
• Updated: 24/Aug/10 21:18
• Resolved: 21/Oct/09 16:31

Description

Currently the CS caps a queue's capacity to it's actual capacity if a hard-limit is specified to be greater than it's actual capacity. We should allow the queue to go upto the hard-limit if specified.

Also, I propose we change the hard-limit unit to be percentage rather than #slots.