[SPARK-50301] TransformWithState metrics incorrectly counts the time for state removal

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 4.0.0
• Fix Version/s: 4.0.0
• Component/s: Structured Streaming
• Labels: pull-request-available
• Assignee: Neil Ramaswamy
• Reporter: Neil Ramaswamy
• Created: 13/Nov/24 19:33
• Updated: Yesterday 05:41
• Resolved: Yesterday 05:31

Description

In TWS, we use the streaming query progress metric `allRemovalsTimeMs` to actually count the amount of time it takes to process the user's timers. This is inconsistent with the other stateful operators, which use `allRemovalsTimeMs` to count the amount of time needed to remove state. The mis-labelling of the metric happens here.

Note that this likely resulted as a copy-paste bug, since `flatMapGroupsWithState` has the same bug here. We should fix that too.

Additionally, TWS uses `commitTimeMs` to capture the amount of time taken to perform state removal due to TTL. This is incorrect; this should actually be part of `allRemovalsTimeMs`.

Issue Links

• links to: GitHub Pull Request #48862