[SPARK-19690] Join a streaming DataFrame with a batch DataFrame may not work

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Duplicate
• Affects Version/s: 2.0.3, 2.1.0, 2.1.1
• Component/s: Structured Streaming
• Target Version/s: 2.3.0
• Assignee: Unassigned
• Reporter: Shixiong Zhu
• Created: 22/Feb/17 01:12
• Updated: 22/Aug/17 02:23
• Resolved: 22/Aug/17 02:23

Description

When joining a streaming DataFrame with a batch DataFrame, if the batch DataFrame has an aggregation, it will be converted to a streaming physical aggregation. Then the query will crash.

Issue Links

• is related to: SPARK-20672 Keep the `isStreaming` property in triggerLogicalPlan in Structured Streaming
• links to: [Github] Pull Request #17025 (uncleGen)
• links to: [Github] Pull Request #17052 (uncleGen)