[SPARK-47733] Add operational metrics for TWS operators

• Type: Task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Epic Link: Structured Streaming - Arbitrary State API v2
• Assignee: Anish Shrigondekar
• Reporter: Jing Zhan
• Created: 04/Apr/24 23:48
• Updated: 12/Apr/24 07:54

Description

Add metrics to improve observability for the newly added operator TransformWithState and some changes we've made into RocksDB.
Proposed metrics to add:

• on the RocksDB StateStore metrics side, we will add the following:

• • num external col families
  • num internal col families
• on the operator side, we will add the following:

• • number of state vars
  • count of state vars by type
  • output mode
  • timeout mode
  • registered timers in batch
  • expired timers in batch
  • initial state enabled or not
  • number of state vars removed in batch

Issue Links

• links to: GitHub Pull Request #45884
• links to: GitHub Pull Request #45937