[SPARK-28652] spark.kubernetes.pyspark.pythonVersion is never passed to executors - ASF JIRA

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Test
Status: Resolved
Priority: Minor
Resolution: Fixed
Affects Version/s: 2.4.3
Fix Version/s: 3.0.0
Component/s: Kubernetes, Spark Core
Labels:
None

Flags:

Important

Description

I suppose this may not be a priority with Python2 on its way out, but given that this setting is only ever sent to the driver and not the executors, no actual work can be performed when the versions don't match, which will tend to be always with the default setting for the driver being changed from 2 to 3, and the executors using `python`, which defaults to v2, by default.