Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-30514

add ENV_PYSPARK_MAJOR_PYTHON_VERSION support for JavaMainAppResource

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: In Progress
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 3.1.0
    • Fix Version/s: None
    • Component/s: Kubernetes, Spark Core
    • Labels:
      None

      Description

      In apache Livy, the program is first started with JavaMainAppResource, and then start the Python worker. At this time, the program needs to be able to pass Python environment variables.

      In spark on yarn, we support it through spark.yarn.isPython. In k8s, we can support this in a better way.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                Jackey Lee Jackey Lee
              • Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                • Created:
                  Updated: