[SPARK-24516] PySpark Bindings for K8S - make Python 3 the default

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.4.0
• Fix Version/s: 3.0.0
• Component/s: Kubernetes, PySpark, Spark Core
• Labels: release-notes
• Target Version/s: 3.0.0
• Assignee: Ilan Filonenko
• Reporter: Ondrej Kokes
• Created: 11/Jun/18 10:07
• Updated: 17/May/20 18:24
• Resolved: 25/Oct/18 06:30

Description

Initial PySpark-k8s bindings have just been resolved (SPARK-23984), but the default Python version there is 2. While you can override this by setting it to 3, I think we should have sensible defaults.

Python 3 has been around for ten years and is the clear successor, Python 2 has only 18 months left in terms of support. There isn't a good reason to suggest Python 2 should be used, not in 2018 and not when both versions are supported.

The relevant commit is here, the version is also in the documentation.

 

Issue Links

• links to: [Github] Pull Request #22810 (ifilonenko)
• links to: [Github] Pull Request #22810 (ifilonenko)