Uploaded image for project: 'Hadoop YARN'
  1. Hadoop YARN
  2. YARN-10496 [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
  3. YARN-10807

Parents node labels are incorrectly added to child queues in weight mode

    XMLWordPrintableJSON

Details

    • Sub-task
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.4.0
    • 3.4.0
    • capacity scheduler
    • None
    • Reviewed

    Description

      In ParentQueue.updateClusterResource when calculating the normalized weights CS will iterate through the parent's nodelabels.
      If the parent has a node label that a specific child doesn't, it will incorrectly add it to the child's node label list through the queueCapacities.setNormalizedWeights(label, weight) call:

      // Normalize weight of children
            if (getCapacityConfigurationTypeForQueues(childQueues)
                == QueueCapacityType.WEIGHT) {
              for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
                float sumOfWeight = 0;
      
                for (CSQueue queue : childQueues) {
                  float weight = Math.max(0,
                      queue.getQueueCapacities().getWeight(nodeLabel));
                  sumOfWeight += weight;
                }
                // When sum of weight == 0, skip setting normalized_weight (so
                // normalized weight will be 0).
                if (Math.abs(sumOfWeight) > 1e-6) {
                  for (CSQueue queue : childQueues) {
                      queue.getQueueCapacities().setNormalizedWeight(nodeLabel,
                          queue.getQueueCapacities().getWeight(nodeLabel) / sumOfWeight);
                  }
                }
              }
            }
      

      Attachments

        1. YARN-10807.001.patch
          1 kB
          Benjamin Teke
        2. YARN-10807.002.patch
          4 kB
          Benjamin Teke

        Activity

          People

            bteke Benjamin Teke
            bteke Benjamin Teke
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: