Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3054

Preempt policy in FairScheduler may cause mapreduce job never finish

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 2.6.0
    • Fix Version/s: None
    • Component/s: fairscheduler
    • Labels:
      None

      Description

      Preemption policy is related with schedule policy now. Using comparator of schedule policy to find preemption candidate cannot guarantee a subset of containers never be preempted. And this may cause tasks to be preempted periodically before they finish. So job cannot make any progress.

      I think preemption in YARN should got below assurance:
      1. Mapreduce jobs can get additional resources when others are idle;
      2. Mapreduce jobs for one user in one queue can still progress with its min share when others preempt resources back.

      Maybe always preempt the latest app and container can get this?

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                peng.zhang Peng Zhang
              • Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: