[BEAM-9282] Create new module for Spark 3 runner

• Type: Sub-task
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.29.0
• Component/s: runner-spark
• Assignee: Ismaël Mejía
• Reporter: Ismaël Mejía
• Created: 10/Feb/20 16:11
• Updated: 30/Mar/21 18:54
• Resolved: 13/Mar/21 12:01

Description

GIven the big amount of different dependencies and some code incompatibilities introduced by the Spark Structured Streaming runner reliance on the unsable Spark Source V2 API it is a good idea to provide independent modules for the Spark 3 runner.

Notice that the idea is still that the users can provide their Spark dependencies.

We should probably rely on a structure similar to the one the Flink runner uses to support their multiple versions.