Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9671

Improve Locality Scheduling when cluster is busy

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Patch Available
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 2.10.0, 3.3.0
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      When a cluster is very busy, scheduling opportunities are few and far between. Scheduling opportunities are how an application knows when to give up looking for decent locality.

      It doesn't make sense to work hard waiting for locality when the odds of it coming are very small and it may actually take a very long time to actually give up.

      This causes the priority of queues to be violated which is the last thing we want to do when the cluster is full.

      Add a mode to disable skipping locality when cluster is busy.

        Attachments

        1. YARN-9671.001.patch
          36 kB
          Muhammad Samir Khan

          Activity

            People

            • Assignee:
              samkhan Muhammad Samir Khan
              Reporter:
              samkhan Muhammad Samir Khan
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: