[YARN-10807] Parents node labels are incorrectly added to child queues in weight mode

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.4.0
• Component/s: capacity scheduler
• Target Version/s: 3.4.0
• Hadoop Flags: Reviewed
• Assignee: Benjamin Teke
• Reporter: Benjamin Teke
• Created: 04/Jun/21 14:43
• Updated: 12/Feb/24 06:05
• Resolved: 08/Jun/21 13:07

Description

In ParentQueue.updateClusterResource when calculating the normalized weights CS will iterate through the parent's nodelabels.
If the parent has a node label that a specific child doesn't, it will incorrectly add it to the child's node label list through the queueCapacities.setNormalizedWeights(label, weight) call:

// Normalize weight of children
      if (getCapacityConfigurationTypeForQueues(childQueues)
          == QueueCapacityType.WEIGHT) {
        for (String nodeLabel : queueCapacities.getExistingNodeLabels()) {
          float sumOfWeight = 0;

          for (CSQueue queue : childQueues) {
            float weight = Math.max(0,
                queue.getQueueCapacities().getWeight(nodeLabel));
            sumOfWeight += weight;
          }
          // When sum of weight == 0, skip setting normalized_weight (so
          // normalized weight will be 0).
          if (Math.abs(sumOfWeight) > 1e-6) {
            for (CSQueue queue : childQueues) {
                queue.getQueueCapacities().setNormalizedWeight(nodeLabel,
                    queue.getQueueCapacities().getWeight(nodeLabel) / sumOfWeight);
            }
          }
        }
      }