[YARN-6492] Generate queue metrics for each partition

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0, 3.3.1
• Fix Version/s: 2.9.3, 3.2.2, 2.10.1, 3.4.0, 3.3.1
• Component/s: capacity scheduler
• Target Version/s: 3.4.0
• Assignee: Manikandan R
• Reporter: Jonathan Hung
• Created: 19/Apr/17 02:32
• Updated: 25/Jan/24 15:54
• Resolved: 01/Jun/20 17:59

Description

We are interested in having queue metrics for all partitions. Right now each queue has one QueueMetrics object which captures metrics either in default partition or across all partitions. (After YARN-6467 it will be in default partition)

But having the partition metrics would be very useful.

Issue Links

• breaks: YARN-10297 TestContinuousScheduling#testFairSchedulerContinuousSchedulingInitTime fails intermittently
• is duplicated by: YARN-9767 PartitionQueueMetrics Issues
• is related to: YARN-9085 Add Guaranteed and MaxCapacity to CSQueueMetrics
• relates to: YARN-6467 CSQueueMetrics needs to update the current metrics for default partition only

Sub-Tasks

• PartitionQueueMetrics Issues: Resolved
• Add QueueMetrics for Custom Resources: Resolved
• Queue metrics turn negative when relabeling a node with running containers to default partition: Open