[BEAM-5495] PipelineResources algorithm is not working in most environments

• Type: Bug
• Status: Triage Needed
• Priority: P2
• Resolution: Fixed
• Fix Version/s: 2.19.0
• Component/s: runner-flink, runner-spark, sdk-java-core
• Labels: Done
• Assignee: Lukasz Gajowy
• Reporter: Romain Manni-Bucau
• Created: 25/Sep/18 07:59
• Updated: 13/Apr/23 11:00
• Resolved: 16/Dec/19 17:28

Description

Issue are:
1. it assumes the classloader is an URLClassLoader (not always true and java >= 9 breaks that as well for the app loader)
2. it uses loader.getURLs() which leads to including the JRE itself in the staged file

Looks like this detect resource algorithm can't work and should be replaced by a SPI rather than a built-in and not extensible algorithm. Another valid alternative is to just drop that "guess" logic and force the user to set staged files.

Issue Links

• blocks: BEAM-2530 Make Beam compatible with next Java LTS version (Java 11)
• relates to: BEAM-7478 Remote cluster submission from Flink Runner broken due to staging issues
• supercedes: BEAM-6656 org.apache.beam.runners.core.construction.PipelineResources#detectClassPathResourcesToStage broken on java > 8
• links to: GitHub Pull Request #10268
• links to: GitHub Pull Request #10410
• links to: GitHub Pull Request #10420
• links to: GitHub Pull Request #10423