Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-45 [Preemption] Scheduler feedback to AM to release containers
  3. YARN-4822

Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.8.0, 3.0.0-alpha1
    • Component/s: None
    • Labels:
      None
    • Target Version/s:
    • Hadoop Flags:
      Reviewed

      Description

      Currently, ProportionalCapacityPreemptionPolicy has hard coded logic to select candidates to be preempted (based on FIFO order of applications/containers). It's not a simple to add new candidate-selection logics, such as preemption for large container, intra-queeu fairness/policy, etc.

      In this JIRA, I propose to do following changes:
      1) Cleanup code bases, consolidate current logic into 3 stages:

      • Compute ideal sharing of queues
      • Select to-be-preempt candidates
      • Send preemption/kill events to scheduler

      2) Add a new interface: PreemptionCandidatesSelectionPolicy for above "select to-be-preempt candidates" part. Move existing how to select candidates logics to FifoPreemptionCandidatesSelectionPolicy.
      3) Allow multiple PreemptionCandidatesSelectionPolicies work together in a chain. Preceding PreemptionCandidatesSelectionPolicy has higher priority to select candidates, and later PreemptionCandidatesSelectionPolicy can make decisions according to already selected candidates and pre-computed queue ideal shares of resources.

        Attachments

        1. YARN-4822.1.patch
          66 kB
          Wangda Tan
        2. YARN-4822.2.patch
          66 kB
          Wangda Tan
        3. YARN-4822.3.patch
          114 kB
          Wangda Tan
        4. YARN-4822.4.patch
          129 kB
          Wangda Tan
        5. YARN-4822.5.patch
          131 kB
          Wangda Tan
        6. YARN-4822.6.patch
          132 kB
          Wangda Tan
        7. YARN-4822.7.patch
          132 kB
          Wangda Tan

          Issue Links

            Activity

              People

              • Assignee:
                leftnoteasy Wangda Tan
                Reporter:
                leftnoteasy Wangda Tan
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: