Simplify configuration related to task-memory-monitoring and memory-based scheduling

• Key: HADOOP-5881
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Vinod K V
• Reporter: Vinod K V
• Fix Version/s: 0.20.1

Description

The configuration we have now is pretty complicated. Besides everything else, the mechanism of not specifying parameters separately for maps and reduces leads to problems like HADOOP-5811. This JIRA should address simplifying things and at the same time solving these problems.

Comments

[Vinod K V]

Some of the problems that stood out and the corresponding solutions, after some discussions with Eric, Arun and Hemanth:

• The current configuration system doesn't distinguish memory usage of maps and reduces. In general, reduces are more memory intensive than maps. Also, because of this lack of distinguishing, we use the total memory available on the TT as a shared resource across slot types. This lead to problems mentioned in HADOOP-5811. The solution is to divide memory resource between map slots and reduce slots. In presence of high memory jobs, map slots of these jobs will use memory of other map slots and don't take over memory from reduce slots. And vice versa. To reflect the differences in general usage between map slots and reduce slots, we may want to specify more map slots on a node than the number of reduce slots.

• We now have separate configuration for specifying default values for the job's memory configuration. This is unnecessary and can better be handled by using layers of configuration. As Arun suggests, default values can be provided by cluster admins to the users via the configuration files distributed to clients.

• With the current configuration, we have 1) total memory available calculated on a TT, 2) a `reserved` memory on TT for system usage. Because of this mechanism, if a TT has lower memory overall, then we assign lower memory to a single slot. As per the discussions, this seems like a wrong idea. To paraphrase Eric - "A slot is a slot, is a slot. TT will just be configured with the number of slots (map & reduce). " In essential, if a TT has lower memory, the correct scheme is to decrease the number of slots and not the memory per slot. Memory allotted for slot should be more or less same across all the TTs in the cluster.

• We are distinguishing virtual memory used by processes with the physical memory. This seems necessary when considering streaming/pipes tasks. However, "in Java, once the VM hits swap, performance degrades fast, we want to configure the limits based on the physical memory on the machine (not including swap), to avoid thrashing". With this in view, there doesn't seem to any need for distinguishing vmem from physical memory w.r.t configuration. Depending on a site's requirements, the configuration items can reflect whether we want tasks to go beyond physical memory or not.

[Vinod K V]

Here's the proposal:

Parameter	Type	Meaning
mapred.cluster.map.memory.mb	set by admin, cluster-wide	Cluster definition of memory per map slot. The maximum amount of memory, in MB, each map task on a tasktracker can consume.
mapred.cluster.reduce.memory.mb	set by admin, cluster-wide	Cluster definition of memory per reduce slot. The maximum amount of memory, in MB, each reduce task on a tasktracker can consume.
mapred.job.map.memory.mb	set by user, per-job	Job requirement for map tasks. The maximum amount of memory each map task of a job can consume, in MB.
mapred.job.reduce.memory.mb	set by user, per-job	job requirement for reduce tasks. The maximum amount of memory each reduce task of a job can consume, in MB.
mapred.cluster.max.map.memory.mb	set by admin, cluster-wide	Max limit on jobs. The maximum value that can be specified by a user via mapred.job.map.memory.mb, in MB. A job that asks for more than this number will be failed at submission itself.
mapred.cluster.max.reduce.memory.mb	set by admin, cluster-wide	Max limit on jobs. The maximum value that can be specified by a user via mapred.job.reduce.memory.mb, in MB. A job that asks for more than this number will be failed at submission itself.

[Vinod K V]

Patch implementing the proposal.

[Vinod K V]

The uploaded patch (for trunk) passed ant test-patch, run-mapred-tests and test-contrib, leaving aside the usual suspects for failure - TestReduceFetch, TestStreamingBadRecords, TestStreamingExitStatus, TestQueueCapacities and TestJobHistory(which succeeds on a clean workspace but not when run along with all the other tests).

[Vinod K V]

Patch for branch-0.20.

[Hemanth Yamijala]

I've been reviewing this patch offline and already had given comments that are incorporated into the new patch. Based on that check, I am a +1 for the code changes. Will commit this.

[Hemanth Yamijala]

I just committed this to trunk and branch 0.20. Thanks, Vinod !

[Owen O'Malley]

I'm concerned that this went in without an opportunity for review. The patch was uploaded at 5:49am and committed at 6:36am.

I'm also concerned that this incompatibly changes the config variables in the 0.20 branch. What is the rationale for pushing these configuration changes back to 0.20? If they are required, they need a compatability story.

[Hemanth Yamijala]

I had a chat with Owen regarding the objections raised.

I'm concerned that this went in without an opportunity for review. The patch was uploaded at 5:49am and committed at 6:36am.

I apologize for the rushed commit and take responsibility. However, as I explained above I had reviewed the patch for a significant time. The first version which I reviewed is not on JIRA. In retrospect, I think that was a mistake, and will avoid repeating it in future. Sorry !

I'm also concerned that this incompatibly changes the config variables in the 0.20 branch. What is the rationale for pushing these configuration changes back to 0.20? If they are required, they need a compatability story.

There was a misunderstanding on this point. It seemed like in an internal discussion we agreed to this approach. However, we are reverting stand after rethinking. I opened HADOOP-5919 to discuss and provide a backwards compatibility story for this feature. Let's keep the discussion there.

Based on these explanations, can I resolve this bug ?

[Owen O'Malley]

Yes, thanks for addressing my concerns Hemanth.

[Ramya R]

It would be useful if the description of the above memory management parameters is added in mapred-default.xml. Also mapred.tasktracker.vmem.reserved, mapred.task.default.maxvmem and mapred.task.limit.maxvmem has to be removed from the xml.

[Hemanth Yamijala]

It would be useful if the description of the above memory management parameters is added in mapred-default.xml. Also mapred.tasktracker.vmem.reserved, mapred.task.default.maxvmem and mapred.task.limit.maxvmem has to be removed from the xml.

+1 regarding documentation. I filed HADOOP-5957 to track this. But we are planning to introduce backwards compatible options, so removing the configuration parameters from xml will not be necessary. Please see HADOOP-5919.

[Hudson]

Integrated in Hadoop-trunk #863 (See http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/863/)