[SPARK-32465] How do I get the SPARK shuffle monitoring indicator?

• Type: Question
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Affects Version/s: 2.2.1
• Component/s: Shuffle
• Labels: Metrics Monitoring
• Assignee: Unassigned
• Reporter: MOBIN
• Created: 28/Jul/20 09:12
• Updated: 29/Jul/20 19:54
• Resolved: 29/Jul/20 19:54

Description

I want to monitor the spark task through graphite_export and prometheus, I have collected execution, driver cpu and memory related indicators, but there is no shuffle read and shuffle write related indicators, how should I configure it?

metrics.properties:
master.source.jvm.class=org.apache.spark.metrics.source.JvmSource
worker.source.jvm.class=org.apache.spark.metrics.source.JvmSource
driver.source.jvm.class=org.apache.spark.metrics.source.JvmSource
executor.source.jvm.class=org.apache.spark.metrics.source.JvmSource
shuffleService.source.jvm.class=org.apache.spark.metrics.source.JvmSource