[SPARK-24377] Make --py-files work in non pyspark application

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.3.0
• Fix Version/s: 2.4.0
• Component/s: Spark Submit
• Assignee: Saisai Shao
• Reporter: Saisai Shao
• Created: 24/May/18 06:46
• Updated: 12/Dec/22 18:10
• Resolved: 29/May/18 02:49

Description

For some Spark applications, though they're a java program, they require not only jar dependencies, but also python dependencies. One example is Livy remote SparkContext application, this application is actually a embedded REPL for Scala/Python/R, so it will not only load in jar dependencies, but also python and R deps.

Currently for a Spark application, --py-files can only be worked for a pyspark application, so it will not be worked in the above case. So here propose to remove such restriction.

Also we tested that "spark.submit.pyFiles" only supports quite limited scenario (client mode with local deps), so here also expand the usage of "spark.submit.pyFiles" to be alternative of --py-files.

Issue Links

• links to: [Github] Pull Request #21420 (jerryshao)