job-level configurable mapred.map.tasks.maximum and mapred.reduce.tasks.maximum

• Key: HADOOP-4295
• Type: Improvement
• Status: Closed
• Resolution: Won't Fix
• Priority: Major
• Assignee: Unassigned
• Reporter: Christian Kunz

Description

Right now mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum are set on the tasktracker level.

In absense of a smart tasktracker monitoring resources and deciding in an adaptive manner how many tasks can be run simultaneously, it would be nice to move these two configuration options to the job level. This would make it easier to optimize the performance of a batch of jobs.

Comments

[Doug Cutting]

I think these are appropriately set at the tasktracker level, since they're meant to correspond to the resources of the tasktracker, e.g., the number of cores. If one has a mixed cluster, with some 2-core nodes and some 4-core nodes, then one might reasonably set these differently on different nodes. The memory limits of HADOOP-2765 and HADOOP-4035 can be used to control things on a per-job basis.

[Christian Kunz]

The situation becomes more complicated when some applications in a batch are pipes applications, some are not. Among pipes applications some might produce a large amount of data to shuffle requiring the java tasks to sort intensively, some not.
In summary, the mapping of number of cores to mapred.map.tasks.maximum and mapred.reduce.tasks.maximum is not always straight forward.

[Doug Cutting]

I'm not arguing that these are perfect, but permitting them to vary per node is a feature that we shouldn't toss out. Adding a different parameter that limits the number of tasks that a job would actually run simultaneously on a node might be reasonable. Thus I think extending the scheduler, as is done in HADOOP-4035, is more like what we'd want here rather than to change these existing parameters.

[Christian Kunz]

Okay, my bad. I went too far by requesting to move the configuration parameters to job-level instead of just adding job-level control

Should the title be changed to something like
Modify the capacity scheduler (HADOOP-3445) to take job limitations concerning number of simultaneous tasks per node into account when scheduling tasks?

[Hemanth Yamijala]

Should the title be changed to something like

Modify the capacity scheduler (HADOOP-3445) to take job limitations concerning number of simultaneous tasks per node into account when scheduling tasks?

I am not sure if Doug was suggesting we use HADOOP-4035 to implement the functionality proposed in this JIRA. I understood it to mean that the approach should be the same. Either way, I think it would be nice to have it handled separately, since HADOOP-4035 is specifically addressing only memory based parameters in job control.

That said, I also think we'll need to consider unifying mechanisms of resource management at some time (maybe in the near future, smile). We already seem to have slightly different ways of dealing with cores, memory, and disk (a.k.a HADOOP-657) - specifying, measuring, reporting and scheduling.

[Christian Kunz]

I am not sure if Doug was suggesting we use HADOOP-4035 to implement the functionality proposed in this JIRA. I understood it to mean that the approach should be the same.

This was my understanding as well.

That said, I also think we'll need to consider unifying mechanisms of resource management at some time (maybe in the near future, smile).

The sooner, the better, smile. Currently one has to restart the framework when mapred.tasktracker.map.tasks.maximum and mapred.tasktracker.reduce.tasks.maximum get changed.

[Vinod K V]

Then, may be, similar to the configuration knob mapred.tasks.maxmemory w.r.t memory, we can have mapred.job.{map|reduce}.tasks to specify number of tasks a job occupies; while mapred.tasktracker.tasks.maxmemory maps to mapred.tasktracker.{map|reduce}.tasks.maximium. After that, similar to how HADOOP-4035 wishes to proceed, a scheduler can compare the job's requirements of number of tasks with tasktracker's limits and scheduler accordingly.

Notes:

• May we should use the term "cores" in mapred.tasktracker.{map|reduce}.tasks.maximium. We clearly need to redefine and distinguish tasks, slots and cores, once and for ever.
• Should we also rename mapred.tasks.maxmemory to mapred.job.tasks.maxmemory?

[Christian Kunz]

I talked with Sameer offline and we agreed to use a work-around based on the scheduler till a more general solution for resource monitoring and utilization is available.