|
If you were logged in you would be able to see more operations.
|
|
|
| Resolution Date: |
16/Dec/08 09:56 AM
|
In today's logic of finding a new task, we assign only one task per heartbeat.
We probably could give the tasktracker multiple tasks subject to the max number of free slots it has - for maps we could assign it data local tasks. We could probably run some logic to decide what to give it if we run out of data local tasks (e.g., tasks from overloaded racks, tasks that have least locality, etc.). In addition to maps, if it has reduce slots free, we could give it reduce task(s) as well. Again for reduces we could probably run some logic to give more tasks to nodes that are closer to nodes running most maps (assuming data generated is proportional to the number of maps). For e.g., if rack1 has 70% of the input splits, and we know that most maps are data/rack local, we try to schedule ~70% of the reducers there.
Thoughts?
|
|
Description
|
In today's logic of finding a new task, we assign only one task per heartbeat.
We probably could give the tasktracker multiple tasks subject to the max number of free slots it has - for maps we could assign it data local tasks. We could probably run some logic to decide what to give it if we run out of data local tasks (e.g., tasks from overloaded racks, tasks that have least locality, etc.). In addition to maps, if it has reduce slots free, we could give it reduce task(s) as well. Again for reduces we could probably run some logic to give more tasks to nodes that are closer to nodes running most maps (assuming data generated is proportional to the number of maps). For e.g., if rack1 has 70% of the input splits, and we know that most maps are data/rack local, we try to schedule ~70% of the reducers there.
Thoughts? |
Show » |
made changes - 31/Mar/08 01:08 PM
| Field |
Original Value |
New Value |
|
Description
|
In today's logic of finding a new task, we assign only one task per heartbeat.
We probably could give the tasktracker multiple tasks subject to the max number of free slots it has - for maps we could assign it data local tasks. We could probably run some logic to decide what to give it if we run out of data local tasks (e.g., tasks from overloaded racks, tasks that have least locality, etc.). In addition to maps, if it has reduce slots free, we could give it reduce task(s) as well. Again for reduces we could probably run some logic to give more tasks to nodes that are closer to nodes running most maps (assuming data generated is proportional to the number of maps). For e.g., if rack1 has 70% of the input splits, we try to schedule ~70% of the reducers there.
Thoughts?
|
In today's logic of finding a new task, we assign only one task per heartbeat.
We probably could give the tasktracker multiple tasks subject to the max number of free slots it has - for maps we could assign it data local tasks. We could probably run some logic to decide what to give it if we run out of data local tasks (e.g., tasks from overloaded racks, tasks that have least locality, etc.). In addition to maps, if it has reduce slots free, we could give it reduce task(s) as well. Again for reduces we could probably run some logic to give more tasks to nodes that are closer to nodes running most maps (assuming data generated is proportional to the number of maps). For e.g., if rack1 has 70% of the input splits, and we know that most maps are data/rack local, we try to schedule ~70% of the reducers there.
Thoughts?
|
made changes - 07/Jun/08 01:26 AM
|
Fix Version/s
|
0.18.0
[ 12312972
]
|
|
made changes - 29/Jul/08 05:00 AM
|
Assignee
|
|
Arun C Murthy
[ acmurthy
]
|
made changes - 29/Jul/08 05:00 AM
|
Fix Version/s
|
|
0.19.0
[ 12313211
]
|
made changes - 05/Aug/08 09:47 AM
made changes - 09/Aug/08 07:30 PM
made changes - 09/Aug/08 07:30 PM
|
Status
|
Open
[ 1
]
|
Patch Available
[ 10002
]
|
made changes - 25/Aug/08 05:05 PM
|
Status
|
Patch Available
[ 10002
]
|
Open
[ 1
]
|
made changes - 12/Sep/08 05:25 AM
made changes - 12/Sep/08 05:25 AM
|
Status
|
Open
[ 1
]
|
Patch Available
[ 10002
]
|
made changes - 15/Sep/08 08:51 AM
|
Status
|
Patch Available
[ 10002
]
|
Open
[ 1
]
|
made changes - 22/Sep/08 08:03 PM
|
Fix Version/s
|
0.19.0
[ 12313211
]
|
|
made changes - 23/Sep/08 09:18 PM
|
Fix Version/s
|
|
0.20.0
[ 12313438
]
|
made changes - 12/Dec/08 09:44 AM
made changes - 13/Dec/08 12:48 AM
made changes - 13/Dec/08 12:48 AM
|
Status
|
Open
[ 1
]
|
Patch Available
[ 10002
]
|
made changes - 15/Dec/08 10:40 AM
|
Status
|
Patch Available
[ 10002
]
|
Open
[ 1
]
|
made changes - 15/Dec/08 07:51 PM
|
Fix Version/s
|
0.20.0
[ 12313438
]
|
|
made changes - 15/Dec/08 11:36 PM
made changes - 16/Dec/08 09:32 AM
|
Fix Version/s
|
|
0.20.0
[ 12313438
]
|
|
Status
|
Open
[ 1
]
|
Patch Available
[ 10002
]
|
made changes - 16/Dec/08 09:56 AM
|
Status
|
Patch Available
[ 10002
]
|
Resolved
[ 5
]
|
|
Resolution
|
|
Fixed
[ 1
]
|
made changes - 23/Apr/09 07:17 PM
|
Status
|
Resolved
[ 5
]
|
Closed
[ 6
]
|
made changes - 08/Jul/09 04:52 PM
|
Component/s
|
mapred
[ 12310690
]
|
|
|
Improvement
Closed
Major
Assign multiple tasks per TaskTracker heartbeat
