[BEAM-7878] Refine Spark runner dependencies

• Type: Improvement
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.15.0
• Component/s: runner-spark
• Assignee: Ismaël Mejía
• Reporter: Ismaël Mejía
• Created: 01/Aug/19 21:19
• Updated: 24/Jul/20 20:01
• Resolved: 12/Aug/19 09:33

Description

The Spark runner has more dependencies than it needs:

• The jackson_module_scala dependency is marked as compile but it is only needed at runtime so it ends up leaking this module and breaking compatibility for users providing Spark on scala 2.12.
• The dropwizard, scala-library and commons-compress dependencies are declared but unused.
• The Kafka dependency version is not aligned with the rest of Beam.
• The Kafka and zookeeper dependencies are used only for the tests but currently are provided

Issue Links

• links to: GitHub Pull Request #9225
• links to: GitHub Pull Request #9310