[SPARK-22547] Don't include executor ID in metrics name

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Li Haoyi
• Created: 17/Nov/17 18:35
• Updated: 21/May/19 04:16
• Resolved: 21/May/19 04:16

Description

Spark's metrics system prefixes all metrics collected from executors with the executor ID.

• https://github.com/apache/spark/blob/fccb337f9d1e44a83cfcc00ce33eae1fad367695/core/src/main/scala/org/apache/spark/metrics/MetricsSystem.scala#L136

This behavior causes two problems:

• it's not possible to aggregate over executors (since the metric name is different for each host)
• upstream metrics systems like Ganglia or Prometheus are put under high load because of the number of time series to store.

By removing the `executorId` from the name of the metric we register, that solves both the above problems