[MAPREDUCE-1859] maxConcurrentMapTask & maxConcurrentReduceTask per job

• Type: New Feature
• Status: Resolved
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 0.20.2
• Component/s: job submission
• Assignee: Unassigned
• Reporter: Johannes Zillmann
• Created: 11/Jun/10 11:10
• Updated: 30/Jul/14 20:14
• Resolved: 30/Jul/14 20:14

Description

It would be valuable if one could specify the max number of map/reduce slots which should be used for a given job. An example would be an map-reduce job importing from a database where you don't want 50 map tasks querying one db at a time but also you don't want to shrink the overall map task count.
Also this is probably already possible through Fair/Capacity-Scheduler or an own Extension i think it would be a good addition for the default TaskScheduler since this seems to be more then a rare used feature.
This would have the benefit in situations where you don't have control/ownership over the cluster as well.
And its more job-centric whereas the existing scheduler extensions seems to be more job-type-centric.

Implementing this feature should be relatively straightforward. Adding something like jobConf.setMaxConcurrentMapTask(int) and respecting this configuration in JobQueueTaskScheduler.

Not sure if this feature would be harmonical with the existing Fair/Capacity-Schedulers.