Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3054

Preempt policy in FairScheduler may cause mapreduce job never finish

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Duplicate
    • 2.6.0
    • None
    • fairscheduler
    • None

    Description

      Preemption policy is related with schedule policy now. Using comparator of schedule policy to find preemption candidate cannot guarantee a subset of containers never be preempted. And this may cause tasks to be preempted periodically before they finish. So job cannot make any progress.

      I think preemption in YARN should got below assurance:
      1. Mapreduce jobs can get additional resources when others are idle;
      2. Mapreduce jobs for one user in one queue can still progress with its min share when others preempt resources back.

      Maybe always preempt the latest app and container can get this?

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              peng.zhang Peng Zhang
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: