Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-9050 [Umbrella] Usability improvements for scheduler activities
  3. YARN-9623

Auto adjust max queue length of app activities to make sure activities on all nodes can be covered

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 3.3.0
    • Component/s: None
    • Labels:
      None
    • Hadoop Flags:
      Reviewed

      Description

      Currently we can use configuration entry "yarn.resourcemanager.activities-manager.app-activities.max-queue-length" to control max queue length of app activities, but in some scenarios , this configuration may need to be updated in a growing cluster. Moreover, it's better for users to ignore that conf therefor it should be auto adjusted internally.
      There are some differences among different scheduling modes:

      • multi-node placement disabled
        • Heartbeat driven scheduling: max queue length of app activities should not less than the number of nodes, considering nodes can not be always in order, we should make some room for misorder, for example, we can guarantee that max queue length should not be less than 1.2 * numNodes
        • Async scheduling: every async scheduling thread goes through all nodes in order, in this mode, we should guarantee that max queue length should be numThreads * numNodes.
      • multi-node placement enabled: activities on all nodes can be involved in a single app allocation, therefor there's no need to adjust for this mode.

      To sum up, we can adjust the max queue length of app activities like this:

      int configuredMaxQueueLength;
      int maxQueueLength;
      serviceInit(){
        ...
        configuredMaxQueueLength = ...; //read configured max queue length
        maxQueueLength = configuredMaxQueueLength; //take configured value as default
      }
      CleanupThread#run(){
        ...
        if (multiNodeDisabled) {
          if (asyncSchedulingEnabled) {
             maxQueueLength = max(configuredMaxQueueLength, numSchedulingThreads * numNodes);
          } else {
             maxQueueLength = max(configuredMaxQueueLength, 1.2 * numNodes);
          }
        } else if (maxQueueLength != configuredMaxQueueLength) {
          maxQueueLength = configuredMaxQueueLength;
        }
      }
      

        Attachments

        1. YARN-9623.001.patch
          13 kB
          Tao Yang
        2. YARN-9623.002.patch
          13 kB
          Tao Yang

          Issue Links

            Activity

              People

              • Assignee:
                Tao Yang Tao Yang
                Reporter:
                Tao Yang Tao Yang
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: