Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-45 [Preemption] Scheduler feedback to AM to release containers
  3. YARN-4822

Refactor existing Preemption Policy of CS for easier adding new approach to select preemption candidates

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 2.8.0, 3.0.0-alpha1
    • None
    • None
    • Reviewed

    Description

      Currently, ProportionalCapacityPreemptionPolicy has hard coded logic to select candidates to be preempted (based on FIFO order of applications/containers). It's not a simple to add new candidate-selection logics, such as preemption for large container, intra-queeu fairness/policy, etc.

      In this JIRA, I propose to do following changes:
      1) Cleanup code bases, consolidate current logic into 3 stages:

      • Compute ideal sharing of queues
      • Select to-be-preempt candidates
      • Send preemption/kill events to scheduler

      2) Add a new interface: PreemptionCandidatesSelectionPolicy for above "select to-be-preempt candidates" part. Move existing how to select candidates logics to FifoPreemptionCandidatesSelectionPolicy.
      3) Allow multiple PreemptionCandidatesSelectionPolicies work together in a chain. Preceding PreemptionCandidatesSelectionPolicy has higher priority to select candidates, and later PreemptionCandidatesSelectionPolicy can make decisions according to already selected candidates and pre-computed queue ideal shares of resources.

      Attachments

        1. YARN-4822.1.patch
          66 kB
          Wangda Tan
        2. YARN-4822.2.patch
          66 kB
          Wangda Tan
        3. YARN-4822.3.patch
          114 kB
          Wangda Tan
        4. YARN-4822.4.patch
          129 kB
          Wangda Tan
        5. YARN-4822.5.patch
          131 kB
          Wangda Tan
        6. YARN-4822.6.patch
          132 kB
          Wangda Tan
        7. YARN-4822.7.patch
          132 kB
          Wangda Tan

        Issue Links

          Activity

            People

              leftnoteasy Wangda Tan
              leftnoteasy Wangda Tan
              Votes:
              0 Vote for this issue
              Watchers:
              7 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: