[SPARK-36088] 'spark.archives' does not extract the archive file into the driver under client mode

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.2
• Component/s: Kubernetes, Spark Submit
• Assignee: Unassigned
• Reporter: rickcheng
• Created: 12/Jul/21 09:30
• Updated: 12/Dec/22 18:11

Description

When running spark in the k8s cluster, there are 2 deploy modes: cluster and client. After my test, in the cluster mode, spark.archives can extract the archive file to the working directory of the executors and driver. But in client mode, spark.archives can only extract the archive file to the working directory of the executors.

 

However, I need spark.archives to send the virtual environment tar file packaged by conda to both the driver and executors under client mode (So that the executor and the driver have the same python environment).

 

Why spark.archives does not extract the archive file into the working directory of the driver under client mode?