Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-36088

'spark.archives' does not extract the archive file into the driver under client mode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.2
    • Fix Version/s: None
    • Component/s: Kubernetes, Spark Submit
    • Labels:
      None

      Description

      When running spark in the k8s cluster, there are 2 deploy modes: cluster and client. After my test, in the cluster mode, spark.archives can extract the archive file to the working directory of the executors and driver. But in client mode, spark.archives can only extract the archive file to the working directory of the executors.

       

      However, I need spark.archives to send the virtual environment tar file packaged by conda to both the driver and executors under client mode (So that the executor and the driver have the same python environment).

       

      Why spark.archives does not extract the archive file into the working directory of the driver under client mode?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              rickcheng rickcheng
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated: