[BEAM-12094] Support Spark 3 in spark_runner.py

• Type: Sub-task
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.32.0
• Component/s: runner-spark
• Labels: portability-spark
• Assignee: Kyle Weaver
• Reporter: Kyle Weaver
• Created: 05/Apr/21 16:48
• Updated: 27/Jun/21 14:21
• Resolved: 27/Jun/21 14:21

Description

spark_runner.py is the Python wrapper for the Beam Spark runner. It requires the Beam Spark job server jar to operate. The default jar is Spark 2.

To build and use the Spark 3 jar from source:

1. Build it from source with "./gradlew :runners:spark:3:job-server:shadowJar"
2. Use it in Python by passing the pipeline option "--spark_job_server_jar=/.../runners/spark/3/job-server/build/libs/beam-runners-spark-3-job-server-2.x.0-SNAPSHOT.jar"

But this is cumbersome. We should provide an easy way to use the released Spark 3 job server jar.

Issue Links

• links to: GitHub Pull Request #15063