[SPARK-29312] Inconsistent behavior on metrics when dropping rows in state between FlatMapGroupWithState and other stateful operators

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Jungtaek Lim
• Created: 01/Oct/19 09:02
• Updated: 01/Oct/19 09:41

Description

Even rows are evicted from state, we don't increase the value of "updated" metrics. The rule applies on streaming aggregations as well as streaming join, but FlatMapGroupsWithState increases the value of "updated" metrics, allowing the metrics to be shown like (total = 0, updated = 1) which looks weird for who are familiar with metrics with others.