[SPARK-42792] Add support to track FLUSH_WRITE_BYTES for RocksDB state store for streaming

• Type: Task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.4.0
• Fix Version/s: 3.5.0
• Component/s: Structured Streaming
• Assignee: Anish Shrigondekar
• Reporter: Anish Shrigondekar
• Created: 14/Mar/23 21:36
• Updated: 15/Mar/23 06:53
• Resolved: 15/Mar/23 06:53

Description

Add support to track FLUSH_WRITE_BYTES for RocksDB state store for streaming

 

Its useful to get this metric for bytes written during flush from RocksDB as part of the DB custom metrics. We propose to add this to the existing metrics that are collected. There is no additional overhead since we are just querying the internal ticker guage, similar to other metrics.

Issue Links

• links to: [Github] Pull Request #40427 (anishshri-db)