[BEAM-6296] Support Python Spark Runner

• Type: New Feature
• Status: Resolved
• Priority: P2
• Resolution: Duplicate
• Affects Version/s: 2.9.0
• Fix Version/s: Not applicable
• Component/s: runner-spark
• Labels: python
• Assignee: Amit Sela
• Reporter: Lei (Eddy) Xu
• Created: 21/Dec/18 19:21
• Updated: 24/Jul/20 20:02
• Resolved: 27/Dec/18 14:46

Description

Hello, everyone,

It would be great to have a Python version of Spark runner available to Python.

While we are happy of running Apache Beam on Dataflow, there are a few use cases that require different dependencies and OS env which makes it be more appropriate to run on a self-managed Spark cluster. With a spark runner for the python SDK, there will be an option to unify the language to define data pipelines.

Would like to see the community's feedbacks of this feature.

Issue Links

• is duplicated by: BEAM-2891 Spark runs portable pipelines