Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: scheduler
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Implement a FairOrderingPolicy which prefers to allocate to SchedulerProcesses with least current usage, very similar to the FairScheduler's FairSharePolicy.

      The Policy will offer allocations to applications in a queue in order of least resources used, and preempt applications in reverse order (from most resources used). This will include conditional support for sizeBasedWeight style adjustment

      Optionally, based on a conditional configuration to enable sizeBasedWeight (default false), an adjustment to boost larger applications (to offset the natural preference for smaller applications) will adjust the resource usage value based on demand, dividing it by the below value:

      Math.log1p(app memory demand) / Math.log(2);

      In cases where the above is indeterminate (two applications are equal after this comparison), behavior falls back to comparison based on the application id, which is generally lexically FIFO for that comparison

        Attachments

        1. YARN-3319.13.patch
          9 kB
          Craig Welch
        2. YARN-3319.14.patch
          7 kB
          Craig Welch
        3. YARN-3319.17.patch
          10 kB
          Craig Welch
        4. YARN-3319.35.patch
          18 kB
          Craig Welch
        5. YARN-3319.39.patch
          18 kB
          Craig Welch
        6. YARN-3319.45.patch
          10 kB
          Craig Welch
        7. YARN-3319.47.patch
          10 kB
          Craig Welch
        8. YARN-3319.53.patch
          10 kB
          Craig Welch
        9. YARN-3319.58.patch
          13 kB
          Craig Welch
        10. YARN-3319.70.patch
          20 kB
          Craig Welch
        11. YARN-3319.71.patch
          22 kB
          Craig Welch
        12. YARN-3319.72.patch
          22 kB
          Craig Welch
        13. YARN-3319.73.patch
          22 kB
          Craig Welch
        14. YARN-3319.74.patch
          23 kB
          Craig Welch
        15. YARN-3319.75.patch
          23 kB
          Craig Welch

          Issue Links

            Activity

              People

              • Assignee:
                cwelch Craig Welch
                Reporter:
                cwelch Craig Welch
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: