[ZEPPELIN-6030] Bug Report: Apache Zeppelin 0.11.1 Issue with Spark Interpreter

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 0.11.1
• Fix Version/s: 0.11.1
• Component/s: build

Environment

Environment:

• Apache Zeppelin version: 0.11.1
• Operating System: Ubuntu 24.04 LTS
• Java version: OpenJDK 8
• Spark version: 2.4.7
• Hadoop: Fully distributed mode
• YARN: Fully distributed mode
• Zeppelin server and Spark running on different servers
• Spark configured to use YARN in cluster mode

• Flags: Important
• Assignee: Unassigned
• Reporter: onjee
• Created: 08/Jul/24 03:27
• Updated: 24/Jul/24 02:50

Description

Description:

While using Apache Zeppelin version 0.11.1, an issue occurs when running the Spark interpreter. The interpreter fails to initialize properly due to unexpected metadata files that were likely included during the build process on a macOS system. These metadata files cause errors in the initialization of the Spark interpreter, preventing it from functioning correctly.

 

Maybe there's an error in the build.

 

Steps to Reproduce:

1. Set up Apache Zeppelin version 0.11.1.
2. Configure the Spark interpreter.
3. Attempt to run a Spark paragraph.

Expected Behavior: The Spark paragraph should execute without any errors.

Actual Behavior: The Spark interpreter fails to initialize, and the following error messages are logged:

Here, you can see that the JAR file you want to run starts with ._ and it's actually not a JAR file, it's a metadata file for apple mac.

Caused by: java.io.IOException: Error accessing /tmp/hadoop-on5949/nm-local-dir/usercache/on5949/appcache/application_1719905050899_0006/container_1719905050899_0006_01_000001/._spark-scala-2.12-0.11.1.jar at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:190) at scala.reflect.io.FileZipArchive.root$lzycompute(ZipArchive.scala:238) at scala.reflect.io.FileZipArchive.root(ZipArchive.scala:235) at scala.reflect.io.FileZipArchive.allDirs$lzycompute(ZipArchive.scala:272) at scala.reflect.io.FileZipArchive.allDirs(ZipArchive.scala:272) at scala.tools.nsc.classpath.ZipArchiveFileLookup.findDirEntry(ZipArchiveFileLookup.scala:76) at scala.tools.nsc.classpath.ZipArchiveFileLookup.list(ZipArchiveFileLookup.scala:63) at scala.tools.nsc.classpath.ZipArchiveFileLookup.list$(ZipArchiveFileLookup.scala:62) at scala.tools.nsc.classpath.ZipAndJarClassPathFactory$ZipArchiveClassPath.list(ZipAndJarFileLookupFactory.scala:58) at scala.tools.nsc.classpath.AggregateClassPath.$anonfun$list$3(AggregateClassPath.scala:105) ... 40 more Caused by: java.util.zip.ZipException: error in opening zip file at java.util.zip.ZipFile.open(Native Method) at java.util.zip.ZipFile.<init>(ZipFile.java:231) at java.util.zip.ZipFile.<init>(ZipFile.java:157) at java.util.zip.ZipFile.<init>(ZipFile.java:171) at scala.reflect.io.FileZipArchive.scala$reflect$io$FileZipArchive$$openZipFile(ZipArchive.scala:187) ... 49 more
 

 

Additional Information:

• The issue is caused by macOS metadata files prefixed with '._' being included in the Spark interpreter directory. (I don't think this metadata file was in the previous version, so I think it went in by mistake.)

References:

• Stack Overflow discussion on similar issue
• My Velog post detailing the bug and solution(Korean)
• Where I downloaded Zeppelin