Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9730

Support forcing configured partitions to be exclusive based on app node label

    XMLWordPrintableJSON

    Details

      Description

      Use case: queue X has all of its workload in non-default (exclusive) partition P (by setting app submission context's node label set to P). Node in partition Q != P heartbeats to RM. Capacity scheduler loops through every application in X, and every scheduler key in this application, and fails to allocate each time since the app's requested label and the node's label don't match. This causes huge performance degradation when number of apps in X is large.

      To fix the issue, allow RM to configure partitions as "forced-exclusive". If partition P is "forced-exclusive", then:

      • 1a. If app sets its submission context's node label to P, all its resource requests will be overridden to P
      • 1b. If app sets its submission context's node label Q, any of its resource requests whose labels are P will be overridden to Q
      • 2. In the scheduler, we add apps with node label expression P to a separate data structure. When a node in partition P heartbeats to scheduler, we only try to schedule apps in this data structure. When a node in partition Q heartbeats to scheduler, we schedule the rest of the apps as normal.

        Attachments

        1. YARN-9730.001.addendum
          1 kB
          Jonathan Hung
        2. YARN-9730.001.patch
          64 kB
          Jonathan Hung
        3. YARN-9730.002.addendum
          1 kB
          Jonathan Hung
        4. YARN-9730.002.patch
          66 kB
          Jonathan Hung
        5. YARN-9730.003.patch
          66 kB
          Jonathan Hung
        6. YARN-9730-branch-2.001.patch
          66 kB
          Jonathan Hung

          Issue Links

            Activity

              People

              • Assignee:
                jhung Jonathan Hung
                Reporter:
                jhung Jonathan Hung
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: