Details
-
Type:
Bug
-
Status: Closed
-
Priority:
Major
-
Resolution: Fixed
-
Affects Version/s: 0.2.0
-
Fix Version/s: 0.3.0
-
Component/s: None
-
Labels:None
Description
The job tracker during my big sort runs only has 80-120 maps running at a time with 2 tasks/node and 179 nodes and a large queue of map tasks. It seems to be caused by the load balancing using the current load rather than the required load to run all of the queued tasks.
Activity
This patch allows map/reduce jobs to be scheduled using the pending load rather than the current load. Also, I have made changes in the JobInProgress to keep a running count of number of running/finished maps/reduces.
Show
Mahadev konar
added a comment - This patch allows map/reduce jobs to be scheduled using the pending load rather than the current load. Also, I have made changes in the JobInProgress to keep a running count of number of running/finished maps/reduces.
I just committed this. Thanks, Mahadev!