[BEAM-1146] Decrease spark runner startup overhead

• Type: Improvement
• Status: Resolved
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: runner-spark
• Assignee: Aviem Zur
• Reporter: Aviem Zur
• Created: 13/Dec/16 15:29
• Updated: 16/May/20 13:38
• Resolved: 02/Jan/17 14:27

Description

BEAM-921 introduced a lazy singleton instantiated once in each JVM (driver & executors) which utilizes reflection to find all subclasses of Source and Coder
While this is beneficial in its own right, the change added about one minute of overhead in spark runner startup time (which cause the first job/stage to take up to a minute).
The change is in class BeamSparkRunnerRegistrator.

Issue Links

• links to: GitHub Pull Request #1674
• links to: GitHub Pull Request #1674