[BEAM-12095] spark_runner.py broken by Spark 3 upgrade.

• Type: Bug
• Status: Resolved
• Priority: P1
• Resolution: Fixed
• Fix Version/s: 2.29.0
• Component/s: runner-spark
• Assignee: Kyle Weaver
• Reporter: Kyle Weaver
• Created: 05/Apr/21 16:52
• Updated: 08/Apr/21 18:20
• Resolved: 08/Apr/21 18:20

Description

The logic that finds the Spark job server jar has been broken by the Spark 3 upgrade, since it assumes the name of the jar corresponds directly to the Gradle target, which is no longer true.

This path is still not documented properly, but some folks out there have already started using it (including TFX).

Issue Links

• is related to: BEAM-7093 Support Spark 3 in Spark runner
• links to: GitHub Pull Request #14429
• links to: GitHub Pull Request #14434
• links to: GitHub Pull Request #14435
• links to: GitHub Pull Request #14448