Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-45 [Preemption] Scheduler feedback to AM to release containers
  3. YARN-567

RM changes to support preemption for FairScheduler and CapacityScheduler

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Closed
    • Major
    • Resolution: Fixed
    • None
    • 2.1.0-beta
    • resourcemanager
    • None

    Description

      A common tradeoff in scheduling jobs is between keeping the cluster busy and enforcing capacity/fairness properties. FairScheduler and CapacityScheduler takes opposite stance on how to achieve this.

      The FairScheduler, leverages task-killing to quickly reclaim resources from currently running jobs and redistributing them among new jobs, thus keeping the cluster busy but waste useful work. The CapacityScheduler is typically tuned
      to limit the portion of the cluster used by each queue so that the likelihood of violating capacity is low, thus never wasting work, but risking to keep the cluster underutilized or have jobs waiting to obtain their rightful capacity.

      By introducing the notion of a work-preserving preemption we can remove this tradeoff. This requires a protocol for preemption (YARN-45), and ApplicationMasters that can answer to preemption efficiently (e.g., by saving their intermediate state, this will be posted for MapReduce in a separate JIRA soon), together with a scheduler that can issues preemption requests (discussed in separate JIRAs YARN-568 and YARN-569).

      The changes we track with this JIRA are common to FairScheduler and CapacityScheduler, and are mostly propagation of preemption decisions through the ApplicationMastersService.

      Attachments

        1. YARN-567.patch
          4 kB
          Carlo Curino
        2. YARN-567.patch
          4 kB
          Carlo Curino

        Issue Links

          blocks

          Sub-task - The sub-task of the issue YARN-568 FairScheduler: support for work-preserving preemption

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue YARN-569 CapacityScheduler: support for preemption (using a capacity monitor)

          • Major - Major loss of function.
          • Closed
          Is contained by

          New Feature - A new feature of the product, which has yet to be developed. MAPREDUCE-4584 Umbrella: Preemption and restart of MapReduce tasks

          • Major - Major loss of function.
          • Open
          is part of

          Sub-task - The sub-task of the issue YARN-568 FairScheduler: support for work-preserving preemption

          • Major - Major loss of function.
          • Closed
          relates to

          Sub-task - The sub-task of the issue MAPREDUCE-5189 Basic AM changes to support preemption requests (per YARN-45)

          • Major - Major loss of function.
          • Resolved

          Sub-task - The sub-task of the issue MAPREDUCE-5176 Preemptable annotations (to support preemption in MR)

          • Major - Major loss of function.
          • Closed

          Activity

            People

              curino Carlo Curino
              curino Carlo Curino
              Votes:
              0 Vote for this issue
              Watchers:
              13 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: