Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-3891

Apps are starved for resource due to default FairSharePolicy in Fair Scheduler

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Duplicate
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: fairscheduler
    • Labels:
      None

      Description

      Based on FairSharePolicy, FSLeafQueue will try to schedule an app that has the lowest resource consumption first.
      However, we have seen some apps that have several thousand reducers stuck at waiting for some newly scheduled mappers to launch. However, it would be very difficult for such apps to get resources if there are many other jobs with low resource consumption and the queue is full.

      If the queue keeps getting new applications, then the stuck reducers would never finish.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                l201514 Siqi Li
                Reporter:
                l201514 Siqi Li
              • Votes:
                0 Vote for this issue
                Watchers:
                15 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: