[SPARK-19378] StateOperator metrics should still return the total number of rows in state even if there was no data for a trigger

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.1.0
• Fix Version/s: 2.1.1, 2.2.0
• Component/s: Structured Streaming
• Assignee: Burak Yavuz
• Reporter: Burak Yavuz
• Created: 26/Jan/17 23:51
• Updated: 13/Jul/18 05:33
• Resolved: 01/Feb/17 01:14

Description

If you have a StreamingDataFrame with an aggregation, we report a metric called stateOperators which consists of a list of data points per aggregation for our query (With Spark 2.1, only one aggregation is supported).

These data points report:

• numUpdatedStateRows
• numTotalStateRows

If a trigger had no data - therefore was not fired - we return 0 data points, however we should actually return a data point with

• numTotalStateRows: numTotalStateRows in lastExecution
• numUpdatedStateRows: 0

This also affects eventTime statistics. We should still provide the min, max, avg even through the data didn't change.

Issue Links

• links to: [Github] Pull Request #16716 (brkyvz)