OpenJPA
  1. OpenJPA
  2. OPENJPA-1454

InstrumentationFactory crashes the tomcat WebappClassLoader by injecting org.apache.openjpa classes into the SystemClassLoader

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.0-M3
    • Fix Version/s: 2.1.0
    • Component/s: UnenhancedClasses
    • Labels:
      None

      Description

      The InstrumentationFactory tries to find the jar containing the manifest with the pre-main and attaches this jar to the VM.

      This has the side effect that all classes which get loaded via the Instrumentation are added to the 'classes' Vector of the SystemClassLoader. And this causes a problem when running OpenJPA in a WebApplication - because the SystemClassLoader only contains a few of those class definitions, and e.g. executing

      public void setBrokerFactory(BrokerFactory factory)

      { _factory = new DelegatingBrokerFactory(factory, PersistenceExceptions.TRANSLATOR); }

      crashes the WebApp because org.apache.openjpa.util.Exceptions is available in the SystemClassLoader but the dependent J2DoPrivHelper is not.

      This is even more problematic when using openjpa-lib, openjpa-kernel, etc as single jars (instead of openjpa-all), because in this case only openjpa-kernel gets injected into the VMs system classpath - causing the agent to subsequently crash because all the classes from openjpa-lib obviously cannot be found.

      This might also be an explanation to a few other open JIRAs I've read through the last few days.

        Activity

        Hide
        Rick Curtis added a comment -

        Does setting openjpa.DynamicEnhancementAgent=false in your p.xml fix the problem?

        Show
        Rick Curtis added a comment - Does setting openjpa.DynamicEnhancementAgent=false in your p.xml fix the problem?
        Hide
        Mark Struberg added a comment -

        yes, I disabled the whole agent for now (after 2 days of debugging).
        Maybe we should switch instrumentation off by default?

        Show
        Mark Struberg added a comment - yes, I disabled the whole agent for now (after 2 days of debugging). Maybe we should switch instrumentation off by default?
        Hide
        Rick Curtis added a comment -

        Maybe this is a stupid question... but are you using an injected EM/EMF?

        Show
        Rick Curtis added a comment - Maybe this is a stupid question... but are you using an injected EM/EMF?
        Hide
        Mark Struberg added a comment -

        yes and no

        I inject the EntityManager myself via OpenWebBeans, but use an extended EM (Persistence.createEntityManagerFactory())
        The EntityManager gets created via a @Producer, thus lazy initialised in the WebappClassLoader realm.

        Show
        Mark Struberg added a comment - yes and no I inject the EntityManager myself via OpenWebBeans, but use an extended EM (Persistence.createEntityManagerFactory()) The EntityManager gets created via a @Producer, thus lazy initialised in the WebappClassLoader realm.
        Hide
        Rick Curtis added a comment -

        Using buildtime enhancement I assume?

        Show
        Rick Curtis added a comment - Using buildtime enhancement I assume?
        Hide
        Mark Struberg added a comment -

        yes, I use the openjpa-maven-plugin to do all the magic.

        Show
        Mark Struberg added a comment - yes, I use the openjpa-maven-plugin to do all the magic.
        Hide
        Rick Curtis added a comment -

        When this happens is there an exception or anything... ?

        Show
        Rick Curtis added a comment - When this happens is there an exception or anything... ?
        Hide
        Michael Dick added a comment -

        Maybe I'm missing it, but does the problem only occur when using the individual jars, or can this occur when you use openjpa-all?

        Show
        Michael Dick added a comment - Maybe I'm missing it, but does the problem only occur when using the individual jars, or can this occur when you use openjpa-all?
        Hide
        Mark Struberg added a comment -

        Oki, the whole story:

        I used openjpa-all-2.0.0-M3 on tomcat-6.0.20 (+Mojarra-2.0.1 + OpenWebBeans) and my application crashed (It worked on jetty-6 which surprises me a bit).
        I cannot remember seeing any 'Agent failed' or something while using the openjpa-all. But I got funny ClassNotFoundException and NoClassDefFoundErrors. Since our production system is tomcat, I needed to evaluate what's going on.

        So I jumped over and built OpenJPA-2 from trunk and used the single jars. With this setup, both tomcat and jetty crashed
        I then discovered the ClassLoading issue I posted to the list yesterday (ObjectValue#newInstance - will create a Jira). After fixing that part it runs on jetty again.

        Today I digged deeper. Tomcats WebappClassLoader first tries to load the class from the SystemClassLoader to prevent applications from overwriting SE classes, and only if it gets a ClassNotFoundException it uses the war classpath. I then found out that the SystemClassLoader has a few classes of the openjpa-kernel registered (I remember the openjpa.util.Exceptions class). And caused by mechanism in the WebappClassLoader, it tries to use the SystemClassLoader to initialise the whole class tree. Here it epically fails while trying to load classes which are e.g. in openjpa-lib or any other dependency which is not available within the jar which got used to inject the Instrumentation. I remember the J2DoHelper and with openjpa-all I think it was parts of the validator-spec which crashed.

        Since the Instrumentation happens in parallel, it was a huge pita to find out, because the classes which have not been found changed from one server start to the other

        I suspect that the app works with jetty because it doesn't do the SE classes check...

        Show
        Mark Struberg added a comment - Oki, the whole story: I used openjpa-all-2.0.0-M3 on tomcat-6.0.20 (+Mojarra-2.0.1 + OpenWebBeans) and my application crashed (It worked on jetty-6 which surprises me a bit). I cannot remember seeing any 'Agent failed' or something while using the openjpa-all. But I got funny ClassNotFoundException and NoClassDefFoundErrors. Since our production system is tomcat, I needed to evaluate what's going on. So I jumped over and built OpenJPA-2 from trunk and used the single jars. With this setup, both tomcat and jetty crashed I then discovered the ClassLoading issue I posted to the list yesterday (ObjectValue#newInstance - will create a Jira). After fixing that part it runs on jetty again. Today I digged deeper. Tomcats WebappClassLoader first tries to load the class from the SystemClassLoader to prevent applications from overwriting SE classes, and only if it gets a ClassNotFoundException it uses the war classpath. I then found out that the SystemClassLoader has a few classes of the openjpa-kernel registered (I remember the openjpa.util.Exceptions class). And caused by mechanism in the WebappClassLoader, it tries to use the SystemClassLoader to initialise the whole class tree. Here it epically fails while trying to load classes which are e.g. in openjpa-lib or any other dependency which is not available within the jar which got used to inject the Instrumentation. I remember the J2DoHelper and with openjpa-all I think it was parts of the validator-spec which crashed. Since the Instrumentation happens in parallel, it was a huge pita to find out, because the classes which have not been found changed from one server start to the other I suspect that the app works with jetty because it doesn't do the SE classes check...
        Hide
        Mark Struberg added a comment -

        @Rick here is an example stacktrace:

        Caused by: java.lang.ClassNotFoundException: org.apache.openjpa.lib.util.Closeable
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:616)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
        at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1302)
        at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
        at org.apache.openjpa.persistence.EntityManagerFactoryImpl.setBrokerFactory(EntityManagerFactoryImpl.java:95)
        at org.apache.openjpa.persistence.EntityManagerFactoryValue.newFactory(EntityManagerFactoryValue.java:53)
        at org.apache.openjpa.persistence.JPAFacadeHelper.toEntityManagerFactory(JPAFacadeHelper.java:75)
        at org.apache.openjpa.persistence.PersistenceProviderImpl.createEntityManagerFactory(PersistenceProviderImpl.java:109)
        at org.apache.openjpa.persistence.PersistenceProviderImpl.createEntityManagerFactory(PersistenceProviderImpl.java:151)
        at org.apache.openjpa.persistence.PersistenceProviderImpl.createEntityManagerFactory(PersistenceProviderImpl.java:59)
        at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:150)
        at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:70)
        at org.apache.webbeans.resource.spi.se.ResourceServiceImpl.getPersistenceUnit(ResourceServiceImpl.java:111)
        at org.apache.webbeans.resource.spi.se.ResourceServiceImpl.getPersistenceContext(ResourceServiceImpl.java:125)
        at org.apache.webbeans.resource.spi.se.ResourceServiceImpl.getResource(ResourceServiceImpl.java:59)
        at org.apache.webbeans.resource.OpenWebBeansResourcePlugin.injectResources(OpenWebBeansResourcePlugin.java:61)
        at org.apache.webbeans.component.AbstractInjectionTargetBean.postConstructDefault(AbstractInjectionTargetBean.java:230)
        at org.apache.webbeans.component.AbstractInjectionTargetBean.postConstruct(AbstractInjectionTargetBean.java:213)
        at org.apache.webbeans.component.AbstractInjectionTargetBean.afterConstructor(AbstractInjectionTargetBean.java:199)
        at org.apache.webbeans.component.AbstractInjectionTargetBean.createDefaultInstance(AbstractInjectionTargetBean.java:136)

        And this is the Instrumentation part. Please note that only openjpa-kernel gets injected into the root classpath as agent-jar, thus the lib.log.Log causes the clash.

        Exception in thread "Attach Listener" java.lang.NoClassDefFoundError: org/apache/openjpa/lib/log/Log
        at java.lang.Class.getDeclaredMethods0(Native Method)
        at java.lang.Class.privateGetDeclaredMethods(Class.java:2427)
        at java.lang.Class.getMethod0(Class.java:2670)
        at java.lang.Class.getMethod(Class.java:1603)
        at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:294)
        at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:348)
        Caused by: java.lang.ClassNotFoundException: org.apache.openjpa.lib.log.Log
        at java.net.URLClassLoader$1.run(URLClassLoader.java:200)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:303)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:248)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316)
        ... 6 more
        Agent failed to start!

        txs and LieGrue,
        strub

        Show
        Mark Struberg added a comment - @Rick here is an example stacktrace: Caused by: java.lang.ClassNotFoundException: org.apache.openjpa.lib.util.Closeable at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:303) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$000(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:303) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:616) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$000(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:303) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1302) at org.apache.catalina.loader.WebappClassLoader.loadClass(WebappClassLoader.java:1233) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) at org.apache.openjpa.persistence.EntityManagerFactoryImpl.setBrokerFactory(EntityManagerFactoryImpl.java:95) at org.apache.openjpa.persistence.EntityManagerFactoryValue.newFactory(EntityManagerFactoryValue.java:53) at org.apache.openjpa.persistence.JPAFacadeHelper.toEntityManagerFactory(JPAFacadeHelper.java:75) at org.apache.openjpa.persistence.PersistenceProviderImpl.createEntityManagerFactory(PersistenceProviderImpl.java:109) at org.apache.openjpa.persistence.PersistenceProviderImpl.createEntityManagerFactory(PersistenceProviderImpl.java:151) at org.apache.openjpa.persistence.PersistenceProviderImpl.createEntityManagerFactory(PersistenceProviderImpl.java:59) at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:150) at javax.persistence.Persistence.createEntityManagerFactory(Persistence.java:70) at org.apache.webbeans.resource.spi.se.ResourceServiceImpl.getPersistenceUnit(ResourceServiceImpl.java:111) at org.apache.webbeans.resource.spi.se.ResourceServiceImpl.getPersistenceContext(ResourceServiceImpl.java:125) at org.apache.webbeans.resource.spi.se.ResourceServiceImpl.getResource(ResourceServiceImpl.java:59) at org.apache.webbeans.resource.OpenWebBeansResourcePlugin.injectResources(OpenWebBeansResourcePlugin.java:61) at org.apache.webbeans.component.AbstractInjectionTargetBean.postConstructDefault(AbstractInjectionTargetBean.java:230) at org.apache.webbeans.component.AbstractInjectionTargetBean.postConstruct(AbstractInjectionTargetBean.java:213) at org.apache.webbeans.component.AbstractInjectionTargetBean.afterConstructor(AbstractInjectionTargetBean.java:199) at org.apache.webbeans.component.AbstractInjectionTargetBean.createDefaultInstance(AbstractInjectionTargetBean.java:136) And this is the Instrumentation part. Please note that only openjpa-kernel gets injected into the root classpath as agent-jar, thus the lib.log.Log causes the clash. Exception in thread "Attach Listener" java.lang.NoClassDefFoundError: org/apache/openjpa/lib/log/Log at java.lang.Class.getDeclaredMethods0(Native Method) at java.lang.Class.privateGetDeclaredMethods(Class.java:2427) at java.lang.Class.getMethod0(Class.java:2670) at java.lang.Class.getMethod(Class.java:1603) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:294) at sun.instrument.InstrumentationImpl.loadClassAndCallAgentmain(InstrumentationImpl.java:348) Caused by: java.lang.ClassNotFoundException: org.apache.openjpa.lib.log.Log at java.net.URLClassLoader$1.run(URLClassLoader.java:200) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at java.lang.ClassLoader.loadClass(ClassLoader.java:303) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301) at java.lang.ClassLoader.loadClass(ClassLoader.java:248) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:316) ... 6 more Agent failed to start! txs and LieGrue, strub
        Hide
        Rick Curtis added a comment -

        @Mark

        Do you see this same failure if you configure the -javaagent on tomcat/jetty ?

        Show
        Rick Curtis added a comment - @Mark Do you see this same failure if you configure the -javaagent on tomcat/jetty ?
        Hide
        Rick Curtis added a comment -

        @Mark

        Have you had a chance to try this out with the javaagent?

        Show
        Rick Curtis added a comment - @Mark Have you had a chance to try this out with the javaagent?
        Hide
        Donald Woods added a comment -

        My proposed solution is to:
        1) add the validation spec into the openjpa-all jar, to resolve one issue you were seeing
        2) update the user guide to state that the openjpa-all jar should be used with Tomcat when the instrumentation is used, to resolve the issue of missing classes from the system classloader

        Show
        Donald Woods added a comment - My proposed solution is to: 1) add the validation spec into the openjpa-all jar, to resolve one issue you were seeing 2) update the user guide to state that the openjpa-all jar should be used with Tomcat when the instrumentation is used, to resolve the issue of missing classes from the system classloader
        Hide
        Donald Woods added a comment -

        Mark, does the changes introduced by OPENJPA-1410 change the results any? If you use the latest 2.0.0, what happens when you use openjpa-all? Can you ping me offline and possibly provide me with a tarball of your test environment?

        Show
        Donald Woods added a comment - Mark, does the changes introduced by OPENJPA-1410 change the results any? If you use the latest 2.0.0, what happens when you use openjpa-all? Can you ping me offline and possibly provide me with a tarball of your test environment?
        Hide
        Donald Woods added a comment -

        Mark, any updates on this?

        Show
        Donald Woods added a comment - Mark, any updates on this?
        Hide
        Donald Woods added a comment -

        no response since including the bval spec in openjpa-all, so closing. please open a new JIRA if more work is needed.

        Show
        Donald Woods added a comment - no response since including the bval spec in openjpa-all, so closing. please open a new JIRA if more work is needed.

          People

          • Assignee:
            Donald Woods
            Reporter:
            Mark Struberg
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development