[BEAM-7587] Spark portable runner: Streaming mode

• Type: Wish
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: Missing
• Component/s: runner-spark
• Labels: portability-spark
• Assignee: Anna Qin
• Reporter: Kyle Weaver
• Created: 18/Jun/19 18:37
• Updated: 13/Jan/22 21:51
• Resolved: 15/May/21 17:55

Description

So far all work on the Spark portable runner has been in batch mode. This is intended as an uber-issue for tracking progress on adding support for streaming.

It might be advantageous to wait for the structured streaming (non-portable) runner to be completed (to some reasonable extent) before undertaking this, rather than using the DStream API. Since work on the structured streaming runner is blocked by SPARK-26655, we should implement this using DStreams instead.

Issue Links

• links to: GitHub Pull Request #12157