[BEAM-8470] Create a new Spark runner based on Spark Structured streaming framework

• Type: New Feature
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.18.0
• Component/s: runner-spark
• Labels: structured-streaming
• Assignee: Etienne Chauchot
• Reporter: Etienne Chauchot
• Created: 24/Oct/19 13:26
• Updated: 13/Apr/23 10:58
• Resolved: 05/Dec/19 15:09

Description

Why is it worth creating a new runner based on structured streaming:

Because this new framework brings:

• Unified batch and streaming semantics:
• no more RDD/DStream distinction, as in Beam (only PCollection)

• Better state management:
• incremental state instead of saving all each time
• No more synchronous saving delaying computation: per batch and partition delta file saved asynchronously + in-memory hashmap synchronous put/get

• Schemas in datasets:
• The dataset knows the structure of the data (fields) and can optimize later on
• Schemas in PCollection in Beam

• New Source API
• Very close to Beam bounded source and unbounded sources

Why make a new runner from scratch?

• Structured streaming framework is very different from the RDD/Dstream framework

We hope to gain

• More up to date runner in terms of libraries: leverage new features
• Leverage learnt practices from the previous runners
• Better performance thanks to the DAG optimizer (catalyst) and by simplifying the code.
• Simplify the code and ease the maintenance

 

Issue Links

• is duplicated by: BEAM-198 Spark runner batch translator to work with Datasets instead of RDDs
• supercedes: BEAM-913 Create the skeleton for a Dataset API Spark runner
• links to: GitHub Pull Request #9866
• links to: GitHub Pull Request #10170
• links to: GitHub Pull Request #10171
• links to: GitHub Pull Request #10211
• links to: GitHub Pull Request #10221