[SPARK-4160] Standalone cluster mode does not upload all needed jars to driver node

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 1.2.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Marcelo Masiero Vanzin
• Created: 30/Oct/14 23:01
• Updated: 21/May/19 05:36
• Resolved: 21/May/19 05:36

Description

If you look at the code in DriverRunner.scala, there is code to download the main application jar from the launcher node. But that's the only jar that's downloaded - if the driver depends on one of the jars or files specified via spark-submit --jars <list> --files <list>, it won't be able to run.

It should be possible to use the same mechanism to distribute the other files to the driver node, even if that's not the most efficient way of doing it. That way, at least, you don't need any external dependencies to be able to distribute the files.

Issue Links

• is duplicated by: SPARK-17978 --jars option in spark-submit does not load jars for driver in spark - standalone mode
• is duplicated by: SPARK-16364 Allow spark-submit to upload jars to nodes in cluster mode