[SPARK-21597] Avg event time calculated in progress may be wrong

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.1.1, 2.2.0
• Fix Version/s: 2.2.1, 2.3.0
• Component/s: Structured Streaming
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 01/Aug/17 21:45
• Updated: 01/Feb/19 00:02
• Resolved: 02/Aug/17 18:00

Description

Right now it stores sum of event times and uses it to calc avg. However, the sum value may be overflow.

Issue Links

• causes: SPARK-26806 EventTimeStats.merge doesn't handle "zero.merge(zero)" correctly
• links to: GitHub