[SPARK-33375] [CORE] Add config spark.yarn.pyspark.archives

• Type: Improvement
• Status: In Progress
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 3.0.1
• Component/s: Spark Submit
• Assignee: Unassigned
• Reporter: Zhongwei Zhu
• Created: 06/Nov/20 19:44
• Updated: 06/Nov/20 19:58

Description

This config provides similar function as `spark.yarn.archive`, but instead passing location of pyspark.zip and py4j.zip. It has below benefits:

1. If I have different version of `spark.yarn.archive` from current machine, then there is no way to provide matched pyspark version. This happens in version upgrade.
2. This can save effort of uploading pyspark.zip and py4j.zip every time.

Issue Links

• links to: [Github] Pull Request #30282 (warrenzhu25)