[SPARK-31483] pyspark shell IPython launch throws ".../pyspark/bin/load-spark-env.sh: No such file or directory"

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.4.5
• Component/s: PySpark, Spark Shell
• Labels: bulk-closed

Environment

$ uname -a

Darwin mengyu-C02Z7885LVDQ 19.3.0 Darwin Kernel Version 19.3.0: Thu Jan  9 20:58:23 PST 2020; root:xnu-6153.81.5~1/RELEASE_X86_64 x86_64

$ python -V

Python 3.7.7

$ ipython -V

7.13.0

 

• Assignee: Unassigned
• Reporter: Zhang
• Created: 18/Apr/20 22:13
• Updated: 25/May/21 01:50
• Resolved: 25/May/21 01:39

Description

I'm trying launching pyspark shell with IPython interface via

PYSPARK_DRIVER_PYTHON=ipython pyspark

However it hits ".../pyspark/bin/load-spark-env.sh: No such file or directory"

(py3-spark) mengyu@mengyu-C02Z7885LVDQ:~/workspace/tmp$ PYSPARK_DRIVER_PYTHON=ipython pyspark
/Users/mengyu/opt/anaconda2/envs/py3-spark/bin/pyspark: line 24: /Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/load-spark-env.sh: No such file or directory
/Users/mengyu/opt/anaconda2/envs/py3-spark/bin/pyspark: line 77: /Users/mengyu/workspace/tmp//Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/spark-submit: No such file or directory
/Users/mengyu/opt/anaconda2/envs/py3-spark/bin/pyspark: line 77: exec: /Users/mengyu/workspace/tmp//Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/spark-submit: cannot execute: No such file or directory

 

It is strange because the path "/Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/load-spark-env.sh" exists.

 

$ file /Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/load-spark-env.sh/Users/mengyu/opt/anaconda2/envs/py3-spark/lib/python3.7/site-packages/pyspark/bin/load-spark-env.sh: Bourne-Again shell script text executable, ASCII text

 

Issue Links

• links to: GitHub Pull Request #28256