Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-33375

[CORE] Add config spark.yarn.pyspark.archives

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: 3.0.1
    • Fix Version/s: None
    • Component/s: Spark Submit
    • Labels:
      None

      Description

      This config provides similar function as `spark.yarn.archive`, but instead passing location of pyspark.zip and py4j.zip. It has below benefits:

      1. If I have different version of `spark.yarn.archive` from current machine, then there is no way to provide matched pyspark version. This happens in version upgrade.
      2. This can save effort of uploading pyspark.zip and py4j.zip every time.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              warrenzhu25 Zhongwei Zhu
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: