If you have a StreamingDataFrame with an aggregation, we report a metric called stateOperators which consists of a list of data points per aggregation for our query (With Spark 2.1, only one aggregation is supported).
These data points report:
If a trigger had no data - therefore was not fired - we return 0 data points, however we should actually return a data point with
- numTotalStateRows: numTotalStateRows in lastExecution
- numUpdatedStateRows: 0
This also affects eventTime statistics. We should still provide the min, max, avg even through the data didn't change.