Geronimo
  1. Geronimo
  2. GERONIMO-3687

classloader deadlock during server startup

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.1
    • Fix Version/s: 2.0.3, 2.1.1, 2.2
    • Component/s: None
    • Security Level: public (Regular issues)
    • Labels:
      None

      Description

      I've been seeing regular deadlocks running Geronimo on Leopard.

      It's the same basic scenario as found in GERONIMO-3141. For some reason, the work around in 3141 doesn't work for me on Leopard.

      Just another hack and I'm sure we can fix this...

      Here's background info...

      $ ./geronimo.sh run
      Using GERONIMO_BASE: /Users/kevan/geronimo-jetty6-jee5-2.0.2
      Using GERONIMO_HOME: /Users/kevan/geronimo-jetty6-jee5-2.0.2
      Using GERONIMO_TMPDIR: var/temp
      Using JRE_HOME: /System/Library/Frameworks/JavaVM.framework/Versions/CurrentJDK/Home

      You can then send the java process a QUIT signal (kill -3 <pid>) to have java dump out the thread stack traces. The deadlock occurs during a load of an Iterator class (IIRC). The JVM is not well behaved, IMO. Here's the thread stack traces that I get:

      Full thread dump Java HotSpot(TM) Client VM (1.5.0_13-119 mixed mode):

      "Low Memory Detector" daemon prio=5 tid=0x01009d60 nid=0x858800 runnable [0x00000000..0x00000000]

      "CompilerThread0" daemon prio=9 tid=0x01009330 nid=0x857a00 waiting on condition [0x00000000..0x00000000]

      "Signal Dispatcher" daemon prio=9 tid=0x01008e60 nid=0x855e00 waiting on condition [0x00000000..0x00000000]

      "Finalizer" daemon prio=8 tid=0x01007d10 nid=0x81ba00 waiting for monitor entry [0xb0a05000..0xb0a05d90]
      at org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35)
      at sun.instrument.TransformerManager.transform(TransformerManager.java:122)
      at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155)
      at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82)
      at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)

      "Reference Handler" daemon prio=10 tid=0x01007910 nid=0x81a200 in Object.wait() [0xb0984000..0xb0984d90]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x05a735f8> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:474)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      • locked <0x05a735f8> (a java.lang.ref.Reference$Lock)

      "main" prio=5 tid=0x010018b0 nid=0xb0801000 waiting for monitor entry [0xb07ff000..0xb0800188]
      at java.lang.ClassLoader.findBootstrapClass(Native Method)
      at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:308)

      • locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      • locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280)
      • locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374)
      • locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)

      "VM Thread" prio=9 tid=0x01007060 nid=0x809800 runnable

      "VM Periodic Task Thread" prio=9 tid=0x0100aa00 nid=0x859c00 waiting on condition

      "Exception Catcher Thread" prio=10 tid=0x01001b00 nid=0x80ae00 runnable

      Found one Java-level deadlock:
      =============================
      "Finalizer":
      waiting to lock monitor 0x0081b070 (object 0x05a73660, a sun.misc.Launcher$AppClassLoader),
      which is held by "main"
      "main":
      waiting to lock monitor 0x0081b094 (object 0x09584b40, a [[I),
      which is held by "Finalizer"

      Java stack information for the threads listed above:
      ===================================================
      "Finalizer":
      at org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:35)
      at sun.instrument.TransformerManager.transform(TransformerManager.java:122)
      at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155)
      at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:82)
      at java.lang.ref.Finalizer.access$100(Finalizer.java:14)
      at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:160)
      "main":
      at java.lang.ClassLoader.findBootstrapClass(Native Method)
      at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:946)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:308)

      • locked <0x05a75b78> (a sun.misc.Launcher$ExtClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
      • locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:280)
      • locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
        at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:374)
      • locked <0x05a73660> (a sun.misc.Launcher$AppClassLoader)

      Found 1 deadlock.

        Activity

        Hide
        Kevan Miller added a comment -

        From the stacktrace, the deadlock occurs when we're attempting to load a class on line 35 of TransformerCollection:

        for (ClassFileTransformer transformer : transformers)

        This enhanced for statement is equivalent to:
        for (Iterator i = transformer.iterator(); i.hasNext(); )

        { transformer = i.next(); ... }

        So, the class being loaded on the Finalizer thread, when the deadlock occurs, must be an Iterator class. I think if we force a load of the Iterator class during the load of TransformerCollection, we'll be able to avoid this deadlock.

        IMO, the true culprit is the Sun classloader. However, doubt it will be fixed anytime soon.

        Looks like this fixes my problem. I'll commit shortly.

        Show
        Kevan Miller added a comment - From the stacktrace, the deadlock occurs when we're attempting to load a class on line 35 of TransformerCollection: for (ClassFileTransformer transformer : transformers) This enhanced for statement is equivalent to: for (Iterator i = transformer.iterator(); i.hasNext(); ) { transformer = i.next(); ... } So, the class being loaded on the Finalizer thread, when the deadlock occurs, must be an Iterator class. I think if we force a load of the Iterator class during the load of TransformerCollection, we'll be able to avoid this deadlock. IMO, the true culprit is the Sun classloader. However, doubt it will be fixed anytime soon. Looks like this fixes my problem. I'll commit shortly.
        Hide
        Kevan Miller added a comment -

        Working for me. Let me know if there are any problems...

        Show
        Kevan Miller added a comment - Working for me. Let me know if there are any problems...
        Hide
        Kevan Miller added a comment -

        Ugh... I'm hitting this deadlock again...

        /me bangs head on hard object

        I'm also now seeing exact same behavior as someone described on user/dev list. Runs fine when installed on my /Users/kevan directory, but deadlocks when installed on /

        Show
        Kevan Miller added a comment - Ugh... I'm hitting this deadlock again... /me bangs head on hard object I'm also now seeing exact same behavior as someone described on user/dev list. Runs fine when installed on my /Users/kevan directory, but deadlocks when installed on /
        Hide
        Kevan Miller added a comment -

        This is only a problem on Mac's running Leopard, it seems. It's a JSE bug, but would like to think of a work-around. Will keep it open a while longer, because it bugs me...

        Show
        Kevan Miller added a comment - This is only a problem on Mac's running Leopard, it seems. It's a JSE bug, but would like to think of a work-around. Will keep it open a while longer, because it bugs me...
        Hide
        Jarek Gawor added a comment -

        I just ran into this deadlock twice in a week on a linux machine:

        Found one Java-level deadlock:
        =============================
        "RMI TCP Connection(4)-9.42.75.229":
        waiting to lock monitor 0x081278ec (object 0xd1be5980, a sun.misc.Launcher$AppClassLoader),
        which is held by "main"
        "main":
        waiting to lock monitor 0x08127bac (object 0xc9b84a20, a [[I),
        which is held by "RMI TCP Connection(4)-9.42.75.229"

        Java stack information for the threads listed above:
        ===================================================
        "RMI TCP Connection(4)-9.42.75.229":
        at java.util.AbstractList.iterator(AbstractList.java:337)
        at org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:41)
        at sun.instrument.TransformerManager.transform(TransformerManager.java:122)
        at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155)
        at javax.security.auth.Policy.<clinit>(Policy.java:145)
        at javax.security.auth.SubjectDomainCombiner$5.run(SubjectDomainCombiner.java:481)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.SubjectDomainCombiner.compatPolicy(SubjectDomainCombiner.java:477)
        at javax.security.auth.SubjectDomainCombiner.<clinit>(SubjectDomainCombiner.java:47)
        at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:124)
        at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:137)
        at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232)
        at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:181)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:585)
        at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294)
        at sun.rmi.transport.Transport$1.run(Transport.java:153)
        at java.security.AccessController.doPrivileged(Native Method)
        at sun.rmi.transport.Transport.serviceCall(Transport.java:149)
        at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466)
        at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707)
        at java.lang.Thread.run(Thread.java:595)

        "main":
        at java.lang.ClassLoader.findBootstrapClass(Native Method)
        at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:891)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:301)

        • locked <0xd1be8b70> (a sun.misc.Launcher$ExtClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:299)
        • locked <0xd1be5980> (a sun.misc.Launcher$AppClassLoader)
          at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268)
        • locked <0xd1be5980> (a sun.misc.Launcher$AppClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
          at java.lang.ClassLoader.findSystemClass(ClassLoader.java:882)
          at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadOptimizedClass(MultiParentClassLoader.java:385)
        • locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader)
          at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java:278)
        • locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
          at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        • locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader)
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
          at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
          at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
          at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadOptimizedClass(MultiParentClassLoader.java:422)
        • locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader)
          at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java:278)
        • locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
          at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319)
        • locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader)
          at java.lang.ClassLoader.defineClass1(Native Method)
          at java.lang.ClassLoader.defineClass(ClassLoader.java:620)
          at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
          at java.net.URLClassLoader.defineClass(URLClassLoader.java:260)
          at java.net.URLClassLoader.access$100(URLClassLoader.java:56)
          at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
          at java.security.AccessController.doPrivileged(Native Method)
          at java.net.URLClassLoader.findClass(URLClassLoader.java:188)
          at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadOptimizedClass(MultiParentClassLoader.java:422)
        • locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader)
          at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java:278)
        • locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader)
          at java.lang.ClassLoader.loadClass(ClassLoader.java:251)
          at org.apache.geronimo.gbean.runtime.GBeanInstance.<init>(GBeanInstance.java:227)
          at org.apache.geronimo.kernel.basic.BasicKernel.loadGBean(BasicKernel.java:354)
          at org.apache.geronimo.kernel.config.ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:441)
          at org.apache.geronimo.kernel.config.KernelConfigurationManager.start(KernelConfigurationManager.java:187)
          at org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:549)
        • locked <0xd1c8cb28> (a org.apache.geronimo.kernel.config.EditableKernelConfigurationManager)
          at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
          at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
          at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
          at java.lang.reflect.Method.invoke(Method.java:585)
          at org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34)
          at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124)
          at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:832)
          at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57)
          at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35)
          at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96)
          at org.apache.geronimo.kernel.config.EditableConfigurationManager$$EnhancerByCGLIB$$d239c2c5.startConfiguration(<generated>)
          at org.apache.geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java:158)
          at org.apache.geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:79)
          at org.apache.geronimo.kernel.util.MainConfigurationBootstrapper.main(MainConfigurationBootstrapper.java:45)
          at org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67)
          at org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)
        Show
        Jarek Gawor added a comment - I just ran into this deadlock twice in a week on a linux machine: Found one Java-level deadlock: ============================= "RMI TCP Connection(4)-9.42.75.229": waiting to lock monitor 0x081278ec (object 0xd1be5980, a sun.misc.Launcher$AppClassLoader), which is held by "main" "main": waiting to lock monitor 0x08127bac (object 0xc9b84a20, a [[I), which is held by "RMI TCP Connection(4)-9.42.75.229" Java stack information for the threads listed above: =================================================== "RMI TCP Connection(4)-9.42.75.229": at java.util.AbstractList.iterator(AbstractList.java:337) at org.apache.geronimo.transformer.TransformerCollection.transform(TransformerCollection.java:41) at sun.instrument.TransformerManager.transform(TransformerManager.java:122) at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:155) at javax.security.auth.Policy.<clinit>(Policy.java:145) at javax.security.auth.SubjectDomainCombiner$5.run(SubjectDomainCombiner.java:481) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.SubjectDomainCombiner.compatPolicy(SubjectDomainCombiner.java:477) at javax.security.auth.SubjectDomainCombiner.<clinit>(SubjectDomainCombiner.java:47) at javax.management.remote.rmi.RMIConnectionImpl.<init>(RMIConnectionImpl.java:124) at javax.management.remote.rmi.RMIJRMPServerImpl.makeClient(RMIJRMPServerImpl.java:137) at javax.management.remote.rmi.RMIServerImpl.doNewClient(RMIServerImpl.java:232) at javax.management.remote.rmi.RMIServerImpl.newClient(RMIServerImpl.java:181) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) at sun.rmi.transport.Transport$1.run(Transport.java:153) at java.security.AccessController.doPrivileged(Native Method) at sun.rmi.transport.Transport.serviceCall(Transport.java:149) at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:466) at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:707) at java.lang.Thread.run(Thread.java:595) "main": at java.lang.ClassLoader.findBootstrapClass(Native Method) at java.lang.ClassLoader.findBootstrapClass0(ClassLoader.java:891) at java.lang.ClassLoader.loadClass(ClassLoader.java:301) locked <0xd1be8b70> (a sun.misc.Launcher$ExtClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:299) locked <0xd1be5980> (a sun.misc.Launcher$AppClassLoader) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:268) locked <0xd1be5980> (a sun.misc.Launcher$AppClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.findSystemClass(ClassLoader.java:882) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadOptimizedClass(MultiParentClassLoader.java:385) locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java:278) locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadOptimizedClass(MultiParentClassLoader.java:422) locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java:278) locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at java.lang.ClassLoader.loadClassInternal(ClassLoader.java:319) locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at java.lang.ClassLoader.defineClass1(Native Method) at java.lang.ClassLoader.defineClass(ClassLoader.java:620) at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124) at java.net.URLClassLoader.defineClass(URLClassLoader.java:260) at java.net.URLClassLoader.access$100(URLClassLoader.java:56) at java.net.URLClassLoader$1.run(URLClassLoader.java:195) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:188) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadOptimizedClass(MultiParentClassLoader.java:422) locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at org.apache.geronimo.kernel.config.MultiParentClassLoader.loadClass(MultiParentClassLoader.java:278) locked <0xd1e0de80> (a org.apache.geronimo.kernel.config.MultiParentClassLoader) at java.lang.ClassLoader.loadClass(ClassLoader.java:251) at org.apache.geronimo.gbean.runtime.GBeanInstance.<init>(GBeanInstance.java:227) at org.apache.geronimo.kernel.basic.BasicKernel.loadGBean(BasicKernel.java:354) at org.apache.geronimo.kernel.config.ConfigurationUtil.startConfigurationGBeans(ConfigurationUtil.java:441) at org.apache.geronimo.kernel.config.KernelConfigurationManager.start(KernelConfigurationManager.java:187) at org.apache.geronimo.kernel.config.SimpleConfigurationManager.startConfiguration(SimpleConfigurationManager.java:549) locked <0xd1c8cb28> (a org.apache.geronimo.kernel.config.EditableKernelConfigurationManager) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:585) at org.apache.geronimo.gbean.runtime.ReflectionMethodInvoker.invoke(ReflectionMethodInvoker.java:34) at org.apache.geronimo.gbean.runtime.GBeanOperation.invoke(GBeanOperation.java:124) at org.apache.geronimo.gbean.runtime.GBeanInstance.invoke(GBeanInstance.java:832) at org.apache.geronimo.gbean.runtime.RawInvoker.invoke(RawInvoker.java:57) at org.apache.geronimo.kernel.basic.RawOperationInvoker.invoke(RawOperationInvoker.java:35) at org.apache.geronimo.kernel.basic.ProxyMethodInterceptor.intercept(ProxyMethodInterceptor.java:96) at org.apache.geronimo.kernel.config.EditableConfigurationManager$$EnhancerByCGLIB$$d239c2c5.startConfiguration(<generated>) at org.apache.geronimo.system.main.EmbeddedDaemon.doStartup(EmbeddedDaemon.java:158) at org.apache.geronimo.system.main.EmbeddedDaemon.execute(EmbeddedDaemon.java:79) at org.apache.geronimo.kernel.util.MainConfigurationBootstrapper.main(MainConfigurationBootstrapper.java:45) at org.apache.geronimo.cli.AbstractCLI.executeMain(AbstractCLI.java:67) at org.apache.geronimo.cli.daemon.DaemonCLI.main(DaemonCLI.java:30)
        Hide
        Kevan Miller added a comment -

        Interesting. Thanks for the info. If we're seeing this on non-Mac, non-Leopard machines prolly time to dig deeper... What JSE level is that on?

        Show
        Kevan Miller added a comment - Interesting. Thanks for the info. If we're seeing this on non-Mac, non-Leopard machines prolly time to dig deeper... What JSE level is that on?
        Hide
        Jarek Gawor added a comment -

        This is either a really weird coincidence or something has recently changed but I'm running into this deadlock much more frequently. Just got another one with trunk. The machine is using:

        java version "1.5.0_12"
        Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04)
        Java HotSpot(TM) Server VM (build 1.5.0_12-b04, mixed mode)

        Show
        Jarek Gawor added a comment - This is either a really weird coincidence or something has recently changed but I'm running into this deadlock much more frequently. Just got another one with trunk. The machine is using: java version "1.5.0_12" Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_12-b04) Java HotSpot(TM) Server VM (build 1.5.0_12-b04, mixed mode)
        Hide
        Jarek Gawor added a comment -

        Just an update on this problem. I still get this deadlock maybe once per 2 weeks or slightly less. In my case the server is started via geronimo-maven-plugin which polls the server periodically via JMX during start up. So this problem might not be reproduced with geronimo.sh script but might be reproduced with start-server command.

        Show
        Jarek Gawor added a comment - Just an update on this problem. I still get this deadlock maybe once per 2 weeks or slightly less. In my case the server is started via geronimo-maven-plugin which polls the server periodically via JMX during start up. So this problem might not be reproduced with geronimo.sh script but might be reproduced with start-server command.
        Hide
        Kevan Miller added a comment -

        The best I can figure is that some JVM's are no longer using AbstractList.iterator() calls in extended for loops. From stack traces, the deadlock was occurring on the for loop. Since the static initializer (which was invoking AbstractList.iterator()), something different was obviously going on. I changed the static for loop to actually use an extended for loop, e.g.:

        for (ClassFileTransformer transformer : transformers) {
        }

        Pretty much a nothing-up-my-sleeve-rockie hack... but it's fixed my leopard problem. Pretty sure it will fix Jarek's problem, also.

        Show
        Kevan Miller added a comment - The best I can figure is that some JVM's are no longer using AbstractList.iterator() calls in extended for loops. From stack traces, the deadlock was occurring on the for loop. Since the static initializer (which was invoking AbstractList.iterator()), something different was obviously going on. I changed the static for loop to actually use an extended for loop, e.g.: for (ClassFileTransformer transformer : transformers) { } Pretty much a nothing-up-my-sleeve-rockie hack... but it's fixed my leopard problem. Pretty sure it will fix Jarek's problem, also.
        Hide
        Sree Bodapati added a comment -

        I am seeing this issue in 2.0.0.1
        can someone post a patch on 2.0.0.1 or suggest a way to create this patch so I can avoid this same issue.

        Show
        Sree Bodapati added a comment - I am seeing this issue in 2.0.0.1 can someone post a patch on 2.0.0.1 or suggest a way to create this patch so I can avoid this same issue.
        Hide
        Kevan Miller added a comment -

        This file on branches/2.0 contains the fix – https://svn.apache.org/repos/asf/geronimo/server/branches/2.0/modules/geronimo-transformer/src/main/java/org/apache/geronimo/transformer/TransformerCollection.java

        2.0.0.1 is not a Geronimo release number. Assume you're using WAS CE...

        Show
        Kevan Miller added a comment - This file on branches/2.0 contains the fix – https://svn.apache.org/repos/asf/geronimo/server/branches/2.0/modules/geronimo-transformer/src/main/java/org/apache/geronimo/transformer/TransformerCollection.java 2.0.0.1 is not a Geronimo release number. Assume you're using WAS CE...
        Hide
        Shawn Jiang added a comment -

        If you are using WAS CE. I found that this fix had been included in 2.0.0.3 since 2.0.0.1 in this file: http://publib.boulder.ibm.com/wasce/changes/2003/CHANGES.txt

        Additional Apache Geronimo JIRA patches included in WASCE V2.0.0.3:
        -------------------------------------------------------------------
        The following Apache Geronimo JIRA patches were included and more
        information on each can be found at -
        http://issues.apache.org/jira/browse/GERONIMO

        4267 - Upgrade to released Yoko 1.0 artifacts
        4262 - Include patched build of AMQ 4.1.2 to resolve AMQ-1272 security exposure
        4245 - Upgrade to Tomcat 6.0.18 to pickup latest security fixes
        4119 - request.isUserInRole("some-role") always return false after @EJB injection
        4113 - Cannot deploy context specific Valve GBeans on Tomcat
        4099 - Calling isUserInRole wipes out run-as info
        4096 - Remove PortOffset from SMTPPort definition.
        4008 - HTTP GET requests on Provider endpoints might fail sometimes with Axis2
        4006 - Deployment of EJB-based JAX-WS web service with WebServiceContext injection fails
        3965 - Custom LoginModule uses wrong classloader
        3930 - IllegalArgumentException reading Transaction Log
        3814 - NPE in GBeanOverride.
        3687 - classloader deadlock during server startup

        Show
        Shawn Jiang added a comment - If you are using WAS CE. I found that this fix had been included in 2.0.0.3 since 2.0.0.1 in this file: http://publib.boulder.ibm.com/wasce/changes/2003/CHANGES.txt Additional Apache Geronimo JIRA patches included in WASCE V2.0.0.3: ------------------------------------------------------------------- The following Apache Geronimo JIRA patches were included and more information on each can be found at - http://issues.apache.org/jira/browse/GERONIMO 4267 - Upgrade to released Yoko 1.0 artifacts 4262 - Include patched build of AMQ 4.1.2 to resolve AMQ-1272 security exposure 4245 - Upgrade to Tomcat 6.0.18 to pickup latest security fixes 4119 - request.isUserInRole("some-role") always return false after @EJB injection 4113 - Cannot deploy context specific Valve GBeans on Tomcat 4099 - Calling isUserInRole wipes out run-as info 4096 - Remove PortOffset from SMTPPort definition. 4008 - HTTP GET requests on Provider endpoints might fail sometimes with Axis2 4006 - Deployment of EJB-based JAX-WS web service with WebServiceContext injection fails 3965 - Custom LoginModule uses wrong classloader 3930 - IllegalArgumentException reading Transaction Log 3814 - NPE in GBeanOverride. 3687 - classloader deadlock during server startup

          People

          • Assignee:
            Kevan Miller
            Reporter:
            Kevan Miller
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development