[SPARK-24637] Add metrics regarding state and watermark to dropwizard metrics

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 2.4.0
• Component/s: Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 23/Jun/18 12:35
• Updated: 12/Dec/22 18:10
• Resolved: 07/Aug/18 02:12

Description

Though Spark provides option to enable stream metrics into Dropwizard, it only exposes three metrics: "inputRate-total", "processingRate-total", "latency" which are not enough information to operate.

Since Spark already exposes other metrics as well which are valuable for operation perspective, we could pick some and expose them into Dropwizard as well.

Issue Links

• links to: [Github] Pull Request #21622 (HeartSaVioR)