[SPARK-30514] add ENV_PYSPARK_MAJOR_PYTHON_VERSION support for JavaMainAppResource

• Type: Improvement
• Status: In Progress
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 3.1.0
• Component/s: Kubernetes, Spark Core
• Assignee: Unassigned
• Reporter: Jackey Lee
• Created: 15/Jan/20 01:40
• Updated: 17/May/20 18:25

Description

In apache Livy, the program is first started with JavaMainAppResource, and then start the Python worker. At this time, the program needs to be able to pass Python environment variables.

In spark on yarn, we support it through spark.yarn.isPython. In k8s, we can support this in a better way.

Issue Links

• links to: GitHub Pull Request #27215