[SPARK-22017] watermark evaluation with multi-input stream operators is unspecified

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.2.0
• Fix Version/s: 2.3.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Jose Torres
• Created: 14/Sep/17 21:52
• Updated: 16/Sep/18 01:51
• Resolved: 16/Sep/17 04:11

Description

Watermarks are stored as a single value in StreamExecution. If a query has multiple watermark nodes (which can generally only happen with multi input operators like union), a headOption call will arbitrarily pick one to use as the real one. This will happen independently in each batch, possibly leading to strange and undefined behavior.

We should instead choose the minimum from all watermark exec nodes as the query-wide watermark.

Issue Links

• links to: [Github] Pull Request #19239 (joseph-torres)