I see in the patch attached that only one concrete implementation CapBasedLoadManager is done for the LoadManager which in turn doesn't take into account any resource usage. I guess you are planning a proper implementation for this feature regarding fair-share of memory usage in another JIRA.
Some points still not dealt with in this JIRA. I bring about these points so as to know if you are thinking or have already thought anything about this.
- Job configuration about how users specify the resource usage. Some memory related configuration properties are added to the framework while working for memory monitoring on TTs as well as memory usage based scheduling in CapacityTaskScheduler. You may want to reuse some/all of it.
- Capturing the scheduling decisions involved when we are not able to find a task from a Schedulable because of lack of resources on a given TaskTasker.
Regarding the latter, the current patch just returns null, which is similar to the decision CapacityTaskScheduler used to take in previous versions - i.e. block the TT till it can be given a task from the job at the head of the queue/pool. Sometime back, we investigated how this approach works with FairScheduler and realized some important implications. For e.g, because the order of jobs might change significantly in consecutive iterations of FairScheduler, just returning null may not work at all. Eventually we may end up waiting for a long time if significant number of jobs ask for high amount of resources.