Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.22.0
-
None
-
None
-
None
Description
FairScheduler locality delay feature holds the scheduling of jobs until it gets good locality.
This greatly improves the locality of the tasks. Reduce the cost of traffic.
We have observed the following problem on FairScheduler locality delay:
We have some machines have older data and some newly added machines do not have important data.
When these machines send heartbeat, JT scans tasks to find jobs has the right locality.
Often time, these machines will scan all of the tasks of all the jobs and do not get any tasks.
Scanning all the tasks on the JT is very costly. This makes JT very slow.
And these machines often time do not get scheduled. This hurts the cluster utilization.
Any ideas?