[SPARK-48223] PrometheusServlet does not work well with queryName in spark streaming metric names

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.5.1, 3.3.4, 3.4.3
• Component/s: Spark Core
• Assignee: Unassigned
• Reporter: Mason Chen
• Created: 09/May/24 18:51
• Updated: 09/May/24 18:52

Description

PrometheusServlet does not work well with queryName in the metric name.

For driver metrics, the configuration `spark.metrics.namespace` allows the user to remove the app id from the metric name to build reusable dashboards and metric queries in external systems.

Similarly, we need a feature to replace the query name in the spark streaming metric names to make dashboards and metric queries reusable. Ideally, the query name would be generated as a tag, instead of part of the metric name.

This can be enabled if a feature flag is enabled, to not break existing users who are dependent of the existing metric format.