[ZEPPELIN-3743] java.lang.RuntimeException: Multiple py4j files found under /usr/lib/spark/python/lib

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 0.8.0
• Component/s: pySpark, python-interpreter
• Labels: easyfix triaged

Environment

AWS Linux running EMR 5.16.0 

Zeppelin 0.8 is installed using the latest official full bundle.

 

• Assignee: Unassigned
• Reporter: Valery Khamenya
• Created: 22/Aug/18 06:11
• Updated: 22/Aug/18 06:16

Description

The following error is mistakenly reported:

  java.lang.RuntimeException: Multiple py4j files found under /usr/lib/spark/python/lib

The error happens e.g. during the execution of

import plotly

The content of /usr/lib/spark/python/lib :

$ ll  /usr/lib/spark/python/lib
total 580
-rw-r--r-- 1 root root  42437 29. Jun 18:11 py4j-0.10.7-src.zip
-rw-r--r-- 1 root root   1445 29. Jun 18:11 PY4J_LICENSE.txt
lrwxrwxrwx 1 root root     19 20. Aug 14:01 py4j-src.zip -> py4j-0.10.7-src.zip
-rw-r--r-- 1 root root 543385 29. Jun 18:11 pyspark.zip

That is, a symlink is treated as a file.

The line of code containing bug:

https://github.com/apache/zeppelin/blob/001c621c773db837b4c90bbf7868c94f284247ce/spark/interpreter/src/main/java/org/apache/zeppelin/spark/PythonUtils.java#L74

If symlinks are resolved to the explicit path to the targeted file, then there will be only one file.