[HADOOP-4035] Modify the capacity scheduler (HADOOP-3445) to schedule tasks based on memory requirements and task trackers free memory

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 0.19.0
• Fix Version/s: 0.20.0
• Hadoop Flags: Incompatible change, Reviewed
• Release Note: Changed capacity scheduler policy to take note of task memory requirements and task tracker memory availability.
• Assignee: Vinod Kumar Vavilapalli
• Reporter: Hemanth Yamijala
• Created: 27/Aug/08 12:23
• Updated: 02/May/13 02:29
• Resolved: 03/Dec/08 04:33

Description

HADOOP-3759 introduced configuration variables that can be used to specify memory requirements for jobs, and also modified the tasktrackers to report their free memory. The capacity scheduler in HADOOP-3445 should schedule tasks based on these parameters. A task that is scheduled on a TT that uses more than the default amount of memory per slot can be viewed as effectively using more than one slot, as it would decrease the amount of free memory on the TT by more than the default amount while it runs. The scheduler should make the used capacity account for this additional usage while enforcing limits, etc.

Issue Links

• is blocked by: HADOOP-4236 JobTracker.killJob() fails to kill a job if the job is not yet initialized
• is blocked by: HADOOP-4287 [mapred] jobqueue_details.jsp shows negative count of running and waiting reduces with CapacityTaskScheduler.
• is depended upon by: HADOOP-4523 Enhance how memory-intensive user tasks are handled
• is part of: HADOOP-3444 Implementing a Resource Manager (V1) for Hadoop
• is related to: HADOOP-2776 Web interface uses internal hostnames on EC2