Details
-
Documentation
-
Status: Resolved
-
Trivial
-
Resolution: Fixed
-
None
Description
The configuration documentation states that the environment variable PYSPARK_PYTHON has a default value of python:
http://spark.apache.org/docs/latest/configuration.html
In fact, the default is python2.7:
https://github.com/apache/spark/blob/4f60651cbec1b4c9cc2e6d832ace77e89a233f3a/bin/pyspark#L39-L45
The change that introduced this was discussed here:
https://github.com/apache/spark/pull/2651
Would it be possible to highlight this in the documentation?