[BEAM-7837] Add an adapter class to support changes in Spark 3 API

• Type: Sub-task
• Status: Resolved
• Priority: P3
• Resolution: Fixed
• Fix Version/s: 2.15.0
• Component/s: runner-spark
• Assignee: Ismaël Mejía
• Reporter: Ismaël Mejía
• Created: 29/Jul/19 11:52
• Updated: 24/Jul/20 20:00
• Resolved: 30/Jul/19 16:48

Description

As part of the API clean ups of Spark 3, there are some methods that do not work as before. For details.

SPARK-19287 JavaPairRDD flatMapValues requires function returning Iterable, not Iterator
SPARK-25737 Remove JavaSparkContextVarargsWorkaround and standardize union() methods

If we want to support both Spark 2 and Spark 3 with the same runner code base we should probably create an adapter class for API incompatible code between Spark versions.

Issue Links

• links to: GitHub Pull Request #9191