[YARN-9596] QueueMetrics has incorrect metrics when labelled partitions are involved

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.8.0, 3.3.0
• Fix Version/s: 2.10.0, 3.0.4, 3.3.0, 2.8.6, 3.2.1, 2.9.3, 3.1.3
• Component/s: capacity scheduler
• Assignee: Muhammad Samir Khan
• Reporter: Muhammad Samir Khan
• Created: 03/Jun/19 21:45
• Updated: 14/Oct/19 15:38
• Resolved: 31/Jul/19 16:30

Description

After YARN-6467, QueueMetrics should only be tracking metrics for the default partition. However, the metrics are incorrect when labelled partitions are involved.

Steps to reproduce

==============

1. Configure capacity-scheduler.xml with label configuration
2. Add label "test" to cluster and replace label on node1 to be "test"
3. Note down "totalMB" at <resourcemanager.webapp.address:port>/ws/v1/cluster/metrics
4. Start first job on test queue.
5. Start second job on default queue (does not work if the order of two jobs is swapped).
6. While the two applications are running, the "totalMB" at <resourcemanager.webapp.address:port>/ws/v1/cluster/metrics will go down by the amount of MB used by the first job (screenshots attached).

Alternately:

In TestNodeLabelContainerAllocation.testQueueMetricsWithLabelsOnDefaultLabelNode(), add the following line at the end of the test before rm1.close():

CSQueue rootQueue = cs.getRootQueue();
assertEquals(10*GB,
rootQueue.getMetrics().getAvailableMB() + rootQueue.getMetrics().getAllocatedMB());

There are two nodes of 10GB each and only one of them have a non-default label. The test will also fail against 20*GB check.

Issue Links

• is related to: YARN-9702 Backport YARN-5788 to branch-2.8