[YARN-3319] Implement a FairOrderingPolicy

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.8.0, 3.0.0-alpha1
• Component/s: scheduler
• Hadoop Flags: Reviewed
• Assignee: Craig Welch
• Reporter: Craig Welch
• Created: 10/Mar/15 00:47
• Updated: 30/Aug/16 01:26
• Resolved: 23/Apr/15 17:52

Description

Implement a FairOrderingPolicy which prefers to allocate to SchedulerProcesses with least current usage, very similar to the FairScheduler's FairSharePolicy.

The Policy will offer allocations to applications in a queue in order of least resources used, and preempt applications in reverse order (from most resources used). This will include conditional support for sizeBasedWeight style adjustment

Optionally, based on a conditional configuration to enable sizeBasedWeight (default false), an adjustment to boost larger applications (to offset the natural preference for smaller applications) will adjust the resource usage value based on demand, dividing it by the below value:

Math.log1p(app memory demand) / Math.log(2);

In cases where the above is indeterminate (two applications are equal after this comparison), behavior falls back to comparison based on the application id, which is generally lexically FIFO for that comparison

Issue Links

• depends upon: YARN-3318 Create Initial OrderingPolicy Framework and FifoOrderingPolicy
• depends upon: YARN-3463 Integrate OrderingPolicy Framework with CapacityScheduler
• is related to: YARN-3510 Create an extension of ProportionalCapacityPreemptionPolicy which preempts a number of containers from each application in a way which respects fairness