[SPARK-31838] The streaming output mode validator didn't treat union well

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.0.0
• Component/s: Structured Streaming
• Assignee: Unassigned
• Reporter: Xingcan Cui
• Created: 27/May/20 14:57
• Updated: 27/May/20 22:44

Description

UnsupportedOperationChecker.checkForStreaming didn't treat the union operator well (it only assumes a linear query plan). To temporarily fix the broken semantics before we have a complete improvement plan for the output mode, the validator should do the following.

1. Allow multiple aggregations from different branches of a union operator.
2. For complete output mode, disallow a union on streams with and without aggregations.