Details
-
Improvement
-
Status: Resolved
-
Major
-
Resolution: Incomplete
-
1.2.0
-
None
Description
If you look at the code in DriverRunner.scala, there is code to download the main application jar from the launcher node. But that's the only jar that's downloaded - if the driver depends on one of the jars or files specified via spark-submit --jars <list> --files <list>, it won't be able to run.
It should be possible to use the same mechanism to distribute the other files to the driver node, even if that's not the most efficient way of doing it. That way, at least, you don't need any external dependencies to be able to distribute the files.
Attachments
Issue Links
- is duplicated by
-
SPARK-17978 --jars option in spark-submit does not load jars for driver in spark - standalone mode
- Resolved
-
SPARK-16364 Allow spark-submit to upload jars to nodes in cluster mode
- Resolved