Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.7.0
-
None
Description
LIVY-418 introduced a downstream change from SPARK-13081 where PythonInterpreter honors the spark.pyspark.python configuration. But it fails to honor the spark.pyspark.driver.python that SPARK-13081 also introduces.
As a result, it's not possible to use different path on the driver and the executors with Livy, even though it's possible to do so with Spark.
My understanding is that https://github.com/apache/incubator-livy/commit/64d71aa757a93518427f315e908dcf54373948b2 should start with:
val pythonExec = conf.getOption("spark.pyspark.driver.python") .orElse(conf.getOption("spark.pyspark.python")) .orElse(sys.env.get(...
... just like https://github.com/apache/spark/commit/7a9e25c38380e6c62080d62ad38a4830e44fe753