[SPARK-33530] Support --archives option natively

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.1.0
• Fix Version/s: 3.1.0
• Component/s: Spark Core
• Epic Link: Project Zen
• Assignee: Hyukjin Kwon
• Reporter: Hyukjin Kwon
• Created: 24/Nov/20 11:46
• Updated: 12/Dec/22 18:10
• Resolved: 01/Dec/20 04:43

Description

Currently, spark-submit --archives and spark.yarn.dist.archives configuration are only supported in Yarn modes:

spark-submit --help
...
 Spark on YARN only:
  --queue QUEUE_NAME          The YARN queue to submit to (Default: "default").
  --archives ARCHIVES         Comma separated list of archives to be extracted into the
                              working directory of each executor.

This is actually critical for PySpark to support shipping other packages together, see also https://hyukjin-spark.readthedocs.io/en/stable/user_guide/python_packaging.html#using-zipped-virtual-environment.

Due to this missing feature, PySpark cannot support conda env to ship other packages together.

Issue Links

• is related to: SPARK-33615 Make spark.archives working in Kubernates
• links to: [Github] Pull Request #30486 (HyukjinKwon)
• links to: [Github] Pull Request #30486 (HyukjinKwon)
• links to: [Github] Pull Request #30486 (HyukjinKwon)