[MRQL-92] Use outer-joins for incremental queries in Spark streaming mode

• Type: Improvement
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 0.9.8
• Component/s: Run-Time/Spark, Streaming
• Assignee: Leonidas Fegaras
• Reporter: Leonidas Fegaras
• Created: 15/Jul/16 22:14
• Updated: 16/Jul/16 21:31
• Resolved: 16/Jul/16 21:31

Description

Currently, incremental queries use Spark's coGroup to merge the current state with the results of processing the new data in the stream. With this patch, the merge is done with a special outer join that doesn't shuffle the state again (it only shuffles the results from the new data).

Issue Links

• links to: GitHub Pull Request #22