Issue Details (XML | Word | Printable)

Key: HADOOP-5170
Type: New Feature New Feature
Status: Resolved Resolved
Resolution: Won't Fix
Priority: Major Major
Assignee: Matei Zaharia
Reporter: Jonathan Gray
Votes: 9
Watchers: 27
Operations

If you were logged in you would be able to see more operations.
Hadoop Common

Set max map/reduce tasks on a per-job basis, either per-node or cluster-wide

Created: 04/Feb/09 09:23 PM   Updated: 08/Oct/09 06:41 PM
Return to search
Component/s: None
Affects Version/s: None
Fix Version/s: 0.21.0

Time Tracking:
Not Specified

File Attachments:
  Size
Text File Licensed for inclusion in ASF works h5170.patch 2009-07-07 08:37 PM Owen O'Malley 16 kB
Text File Licensed for inclusion in ASF works HADOOP-5170-tasklimits-v3-0.18.3.patch 2009-05-29 07:17 AM Todd Lipcon 22 kB
Text File Licensed for inclusion in ASF works tasklimits-v2.patch 2009-05-16 10:12 PM Matei Zaharia 6 kB
Text File Licensed for inclusion in ASF works tasklimits-v3-0.19.patch 2009-05-27 02:58 AM Jonathan Gray 6 kB
Text File Licensed for inclusion in ASF works tasklimits-v3.patch 2009-05-21 09:59 PM Matei Zaharia 16 kB
Text File Licensed for inclusion in ASF works tasklimits-v4-20.patch 2009-07-01 04:17 AM rahul k singh 15 kB
Text File Licensed for inclusion in ASF works tasklimits-v4.patch 2009-06-01 05:47 PM Matei Zaharia 15 kB
Text File Licensed for inclusion in ASF works tasklimits.patch 2009-05-01 11:55 PM Matei Zaharia 3 kB

Hadoop Flags: Reviewed
Release Note: Job tracker parameters permit setting limits on the number of maps (or reduces) per job and/or per node.
Resolution Date: 09/Jul/09 09:21 PM


 Description  « Hide
There are a number of use cases for being able to do this. The focus of this jira should be on finding what would be the simplest to implement that would satisfy the most use cases.

This could be implemented as either a per-node maximum or a cluster-wide maximum. It seems that for most uses, the former is preferable however either would fulfill the requirements of this jira.

Some of the reasons for allowing this feature (mine and from others on list):

  • I have some very large CPU-bound jobs. I am forced to keep the max map/node limit at 2 or 3 (on a 4 core node) so that I do not starve the Datanode and Regionserver. I have other jobs that are network latency bound and would like to be able to run high numbers of them concurrently on each node. Though I can thread some jobs, there are some use cases that are difficult to thread (scanning from hbase) and there's significant complexity added to the job rather than letting hadoop handle the concurrency.
  • Poor assignment of tasks to nodes creates some situations where you have multiple reducers on a single node but other nodes that received none. A limit of 1 reducer per node for that job would prevent that from happening. (only works with per-node limit)
  • Poor mans MR job virtualization. Since we can limit a jobs resources, this gives much more control in allocating and dividing up resources of a large cluster. (makes most sense w/ cluster-wide limit)


 All   Comments   Work Log   Change History   Subversion Commits      Sort Order: Ascending order - Click to sort in descending order
Matei Zaharia made changes - 01/May/09 11:55 PM
Field Original Value New Value
Attachment tasklimits.patch [ 12407054 ]
Matei Zaharia made changes - 02/May/09 07:53 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Matei Zaharia made changes - 16/May/09 10:12 PM
Attachment tasklimits-v2.patch [ 12408326 ]
Matei Zaharia made changes - 16/May/09 10:12 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Matei Zaharia made changes - 16/May/09 10:12 PM
Assignee Matei Zaharia [ matei ]
Status Open [ 1 ] Patch Available [ 10002 ]
Matei Zaharia made changes - 21/May/09 09:59 PM
Attachment tasklimits-v3.patch [ 12408747 ]
Matei Zaharia made changes - 21/May/09 09:59 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Matei Zaharia made changes - 21/May/09 10:00 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Matei Zaharia made changes - 21/May/09 10:03 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Matei Zaharia made changes - 21/May/09 10:03 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Jonathan Gray made changes - 27/May/09 02:58 AM
Attachment tasklimits-v3-0.19.patch [ 12409111 ]
Todd Lipcon made changes - 29/May/09 07:17 AM
Attachment HADOOP-5170-tasklimits-v3-0.18.3.patch [ 12409349 ]
Matei Zaharia made changes - 01/Jun/09 05:44 PM
Status Patch Available [ 10002 ] Open [ 1 ]
Matei Zaharia made changes - 01/Jun/09 05:47 PM
Attachment tasklimits-v4.patch [ 12409578 ]
Matei Zaharia made changes - 01/Jun/09 05:47 PM
Status Open [ 1 ] Patch Available [ 10002 ]
Devaraj Das made changes - 04/Jun/09 08:35 AM
Resolution Fixed [ 1 ]
Fix Version/s 0.21.0 [ 12313563 ]
Hadoop Flags [Reviewed]
Status Patch Available [ 10002 ] Resolved [ 5 ]
rahul k singh made changes - 01/Jul/09 04:17 AM
Attachment tasklimits-v4-20.patch [ 12412231 ]
Owen O'Malley made changes - 02/Jul/09 08:55 PM
Status Resolved [ 5 ] Reopened [ 4 ]
Resolution Fixed [ 1 ]
Owen O'Malley made changes - 07/Jul/09 08:37 PM
Attachment h5170.patch [ 12412786 ]
Owen O'Malley made changes - 08/Jul/09 04:53 PM
Component/s mapred [ 12310690 ]
Owen O'Malley made changes - 09/Jul/09 09:21 PM
Resolution Won't Fix [ 2 ]
Status Reopened [ 4 ] Resolved [ 5 ]
Robert Chansler made changes - 08/Oct/09 06:34 PM
Release Note Job tracker parameters permit setting limits on the number of maps (or reduces) per job and/or per node.