[SPARK-26806] EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0
• Fix Version/s: 2.2.4, 2.3.3, 2.4.1, 3.0.0
• Component/s: Structured Streaming
• Assignee: Shixiong Zhu
• Reporter: liancheng
• Created: 01/Feb/19 00:00
• Updated: 06/Feb/19 16:49
• Resolved: 01/Feb/19 19:30

Description

Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will make "avg" become "NaN". And whatever gets merged with the result of "zero.merge(zero)", "avg" will still be "NaN". Then finally, "NaN".toLong will return "0" and the user will see the following incorrect report:

"eventTime" : {
    "avg" : "1970-01-01T00:00:00.000Z",
    "max" : "2019-01-31T12:57:00.000Z",
    "min" : "2019-01-30T18:44:04.000Z",
    "watermark" : "1970-01-01T00:00:00.000Z"
  }

This issue was reported by liancheng

Issue Links

• is caused by: SPARK-21597 Avg event time calculated in progress may be wrong
• links to: GitHub Pull Request #23718