[SPARK-28074] [SS] Log warn message on possible correctness issue for multiple stateful operations in single query

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 3.0.0
• Fix Version/s: 3.0.0
• Component/s: Documentation, Structured Streaming
• Assignee: Jungtaek Lim
• Reporter: Jungtaek Lim
• Created: 17/Jun/19 07:03
• Updated: 01/Oct/19 08:17
• Resolved: 30/Sep/19 13:18

Description

Please refer https://lists.apache.org/thread.html/cc6489a19316e7382661d305fabd8c21915e5faf6a928b4869ac2b4a@%3Cdev.spark.apache.org%3E to see rationalization of the issue.

tl;dr. Using multiple stateful operators without fully understanding how global watermark affects the query brings correctness issue.

While we may not want to fully block the operations given some users could leverage it with understanding of global watermark, it would be necessary for end users to warn on such breaking-correctness possibility, and provide alternative(s).

Issue Links

• is related to: SPARK-28094 Multiple left joins or aggregations in one query produce incorrect results
• links to: GitHub Pull Request #24890