[YARN-9457] Integrate custom resource metrics better for FairScheduler

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: fairscheduler
• Assignee: Unassigned
• Reporter: Szilard Nemeth
• Created: 07/Apr/19 08:19
• Updated: 27/Oct/22 07:04

Description

YARN-8842 added org.apache.hadoop.yarn.server.resourcemanager.scheduler.QueueMetricsForCustomResources.
This class stores all metrics data for custom resource types.
A field is there in QueueMetrics to hold an object of this class.

Similarly, YARN-9322 added FSQueueMetricsForCustomResources and added an object of this class to FSQueueMetrics.

This jira is about to investigate how it is possible to integrate QueueMetricsForCustomResources into QueueMetrics and FSQueueMetricsForCustomResources into FSQueueMetrics.
The trick is that the Metrics annotation (org.apache.hadoop.metrics2.annotation.Metric) is used to expose values on JMX.

We need to implement a mechanism where QueueMetrics / FSQueueMetrics classes do contain a field of the custom resource values which is a map of resource names as keys, and longs as values.
This way, we don't need the new classes (QueueMetricsForCustomResources and FSQueueMetricsForCustomResources), the code could be much cleaner and consistent.

The hardest part possibly is to find a way to expose metrics values from a map. We obviously can't use the Metrics annotation so a mechanism is required to expose the values on JMX.
For a quick search, I haven't found any way like this in the code
wilfreds: Are you aware of any way to expose values like this?
Most probably, we need to check how the Metrics annotation is processed, understand the whole flow and check what is the underlying mechanism of the metrics propagation to the JMX interface.