Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-5990

[Umbrella] Follow-up (phase-2) FairScheduler preemption improvements

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      YARN-4752 overhauled FS preemption. This JIRA is to track follow-up minor improvements discussed in the design doc on YARN-4752.

      The reason for creating this is primarily for better release notes around YARN-4752.

        Attachments

        1.
        FairScheduler: Avoid preempting AM containers Sub-task Resolved Yufei Gu
        2.
        Propagate allowPreemptionFrom flag all the way down to the app Sub-task Resolved Yufei Gu
        3.
        Experiment with using PriorityQueue for storing starved applications Sub-task Resolved Yufei Gu
        4.
        FS Preemption is a trickle for severely starved applications Sub-task Resolved Karthik Kambatla
        5.
        FairScheduler might not trigger preemption when using DRF Sub-task Resolved Karthik Kambatla
        6.
        Set UncaughtExceptionHandler for all FairScheduler threads Sub-task Resolved Yufei Gu
        7.
        Verify app starvation under custom preemption thresholds and timeouts Sub-task Open Unassigned
        8.
        Preemption should consider more resource-requests of a starved app if the initial set can not be satisfied Sub-task Open Kai
        9.
        Test whether FS preemption consider child queues over fair share if the parent is under fair share Sub-task Resolved Yufei Gu
        10.
        FairScheduler preemption and update should not run concurrently Sub-task Resolved Tao Jie
        11.
        Expose # of pre-emptions as a queue counter Sub-task Resolved Karthik Kambatla
        12.
        Identifying starved apps does not need the scheduler writelock Sub-task Resolved Karthik Kambatla
        13.
        Metric preemptCall is not used in new preemption Sub-task Resolved Yufei Gu
        14.
        FairScheduler: Reserve preempted resources for corresponding applications Sub-task Resolved Miklos Szegedi
        15.
        Add more restrictions In FairScheduler Preemption Sub-task Patch Available Zhaohui Xin
        16.
        Shuffle nodes when selecting to be preempted nodes Sub-task Patch Available Zhaohui Xin
        17.
        It's not reasonable to decide whether app is starved by fairShare Sub-task Patch Available Zhaohui Xin
        18.
        FairScheduler: FSStarvedApps is not thread safe Sub-task Patch Available Zhaohui Xin

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              kasha Karthik Kambatla
            • Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated: