[MAPREDUCE-961] ResourceAwareLoadManager to dynamically decide new tasks based on current CPU/memory load on TaskTracker(s)

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.22.0
• Component/s: contrib/fair-share
• Tags: fb
• Assignee: Scott Chen
• Reporter: Dhruba Borthakur
• Created: 08/Sep/09 05:22
• Updated: 13/Jan/11 02:29

Description

Design and develop a ResouceAwareLoadManager for the FairShare scheduler that dynamically decides how many maps/reduces to run on a particular machine based on the CPU/Memory/diskIO/network usage in that machine. The amount of resources currently used on each task tracker is being fed into the ResourceAwareLoadManager in real-time via an entity that is external to Hadoop.

Issue Links

• is blocked by: MAPREDUCE-944 Extend FairShare scheduler to fair-share memory usage in the cluster
• is related to: MAPREDUCE-1167 Make ProcfsBasedProcessTree collect rss memory information

Sub-Tasks

• Collecting cpu and memory usage for TaskTrackers: Closed