[SPARK-26806] EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.1, 2.2.2, 2.2.3, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.4.0
• Fix Version/s: 2.2.4, 2.3.3, 2.4.1, 3.0.0
• Component/s: Structured Streaming
• Assignee: Shixiong Zhu
• Reporter: Cheng Lian
• Created: 01/Feb/19 00:00
• Updated: 10/Oct/19 23:24
• Resolved: 01/Feb/19 19:30

Description

Right now, EventTimeStats.merge doesn't handle "zero.merge(zero)". This will make "avg" become "NaN". And whatever gets merged with the result of "zero.merge(zero)", "avg" will still be "NaN". Then finally, "NaN".toLong will return "0" and the user will see the following incorrect report:

"eventTime" : {
    "avg" : "1970-01-01T00:00:00.000Z",
    "max" : "2019-01-31T12:57:00.000Z",
    "min" : "2019-01-30T18:44:04.000Z",
    "watermark" : "1970-01-01T00:00:00.000Z"
  }

Issue Links

• is caused by: SPARK-21597 Avg event time calculated in progress may be wrong
• links to: GitHub Pull Request #23718