[TOREE-355] java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.2.0
• Labels: easyfix
• Environment: Toree 0.2.0.dev1; CDH 5.9's embedded spark-2.0.0-beta2
• Assignee: Luciano Resende
• Reporter: Adrien Lavoillotte
• Created: 30/Nov/16 09:01
• Updated: 14/Mar/18 18:45
• Resolved: 27/Mar/17 23:55

Description

Using Toree 0.2.0.dev1 with CDH's embedded spark 2 did not cause TOREE-327 for me. Instead, I got this error:

Exception in thread "main" java.lang.IncompatibleClassChangeError: class org.clapper.classutil.asm.ASMEmptyVisitor has interface org.objectweb.asm.ClassVisitor as super class
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at org.clapper.classutil.asm.ClassFile$.load(ClassFinderImpl.scala:250)
        at org.clapper.classutil.ClassFinder.org$clapper$classutil$ClassFinder$$classData(ClassFinder.scala:427)
        at org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
        at org.clapper.classutil.ClassFinder$$anonfun$2.apply(ClassFinder.scala:385)
        at scala.collection.immutable.Stream.map(Stream.scala:418)
        at org.clapper.classutil.ClassFinder.processOpenZip(ClassFinder.scala:385)
        at org.clapper.classutil.ClassFinder.processJar(ClassFinder.scala:340)
        at org.clapper.classutil.ClassFinder.findClassesIn(ClassFinder.scala:329)
        at org.clapper.classutil.ClassFinder.find(ClassFinder.scala:320)
        at org.clapper.classutil.ClassFinder.getClasses(ClassFinder.scala:311)
        at org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
        at org.apache.toree.plugins.PluginSearcher$$anonfun$1.apply(PluginSearcher.scala:73)
        at scala.util.Try$.apply(Try.scala:192)
        at org.apache.toree.plugins.PluginSearcher.loadClassMap(PluginSearcher.scala:73)
        at org.apache.toree.plugins.PluginSearcher.internalClassInfo$lzycompute(PluginSearcher.scala:35)
        at org.apache.toree.plugins.PluginSearcher.internalClassInfo(PluginSearcher.scala:34)
        at org.apache.toree.plugins.PluginSearcher.internal$lzycompute(PluginSearcher.scala:38)
        at org.apache.toree.plugins.PluginSearcher.internal(PluginSearcher.scala:38)
        at org.apache.toree.plugins.PluginManager.internalPlugins$lzycompute(PluginManager.scala:45)
        at org.apache.toree.plugins.PluginManager.internalPlugins(PluginManager.scala:44)
        at org.apache.toree.plugins.PluginManager.initialize(PluginManager.scala:80)
        at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializePlugins(ComponentInitialization.scala:221)
        at org.apache.toree.boot.layer.StandardComponentInitialization$class.initializeComponents(ComponentInitialization.scala:86)
        at org.apache.toree.Main$$anon$1.initializeComponents(Main.scala:35)
        at org.apache.toree.boot.KernelBootstrap.initialize(KernelBootstrap.scala:101)
        at org.apache.toree.Main$.delayedEndpoint$org$apache$toree$Main$1(Main.scala:40)
        at org.apache.toree.Main$delayedInit$body.apply(Main.scala:24)

Apparently in some much older version of ASM (3), ClassVisitor was an interface and not an abstract class. Said much older version can be found in some Hadoop libs or older Spark libs, and ends up in the classpath with CDH's spark 2.

The simplest solution is to shadow ASM, e.g. using JarJAr Links. The following produces an assembly jar that does not have this issue:

echo 'rule org.objectweb.asm.** org.apache.toree.shadow.@0' >relocate.rules
java -jar jarjar-1.4.jar process relocate.rules toree.jar toree-relocated.jar

ASM is a dependency of ClassUtil, for which an issue on this is currently open, but the author says he may not want to shadow ASM, so I thought maybe you might want to.

Issue Links

• is depended upon by: TOREE-369 Enable Toree to run in Yarn cluster mode
• links to: GitHub Pull Request #115
• links to: https://github.com/apache/incubator-toree/pull/115