Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 5.5.0
    • Fix Version/s: NEEDS_REVIEW
    • Component/s: None
    • Labels:
      None
    • Environment:

      Tomcat 6.0.29, Spring 3.0.5, Oracle Java 6, Centos 5

      Description

      We are running a queue to do asynch audit updates, configured via Spring 3.0.5.

      When the queue fills up Tomcat locks up with all catalina threads waiting on an object monitor in Spring. This object monitor is held by the "ActiveMQ Connection Executor: vm://localhost#2986" thread which is itself blocked for ever awaiting the stopped CountDownLatch at TransportConnection.stop(TransportConnection.java:930).

      There are no "ActiveMQ Task" threads running, which suggests that either the task created by stopAsync has completed or did not run.

      A code review leaves us baffled as to how this latch cannot have counted down, but it hasn't. Could Tomcat possibly be silently discarding the thread that was meant to do the stop without throwing an exception?! It seems unlikely but (as I understand it) TaskRunnerFactory is breaking the Servlet spec by running up its own Threads.

      1. JStack Output.rtf
        7 kB
        Robert Elliot
      2. jmsMessageQueues.xml
        3 kB
        Robert Elliot

        Activity

        Hide
        Rob Davies added a comment -

        This is probably fixed in 5.6 - but without a junit test case its difficult to know.

        Show
        Rob Davies added a comment - This is probably fixed in 5.6 - but without a junit test case its difficult to know.
        Hide
        Timothy Bish added a comment -

        Have you tested against 5.6.0 release? The NullPointerException errors were fixed in that release.

        Show
        Timothy Bish added a comment - Have you tested against 5.6.0 release? The NullPointerException errors were fixed in that release.
        Hide
        Jess added a comment -

        Actually, the NPE stacktrace is a bit different:

        java.lang.NullPointerException
        at org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:510)
        at org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:105)
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:316)
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:180)
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
        at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
        at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:217)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:199)
        at java.lang.Thread.run(Thread.java:736)

        Show
        Jess added a comment - Actually, the NPE stacktrace is a bit different: java.lang.NullPointerException at org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:510) at org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:105) at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:316) at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:180) at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69) at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:227) at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:217) at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:199) at java.lang.Thread.run(Thread.java:736)
        Hide
        Jess added a comment -

        I am seeing the same NullPointerException stacktrace from above. Threads are hanging at:

        at java.lang.Object.wait(Native Method)
        at java.lang.Object.wait(Object.java:196)
        at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:516)
        at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40)
        at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60)
        at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1266)
        at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1260)
        at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:250)
        at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:319)
        at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37)
        at java.lang.reflect.Method.invoke(Method.java:611)
        at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:550)
        at $Proxy77.createSession(Unknown Source)
        at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196)
        at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:457)
        at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)

        Show
        Jess added a comment - I am seeing the same NullPointerException stacktrace from above. Threads are hanging at: at java.lang.Object.wait(Native Method) at java.lang.Object.wait(Object.java:196) at org.apache.activemq.transport.failover.FailoverTransport.oneway(FailoverTransport.java:516) at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:40) at org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelator.java:60) at org.apache.activemq.ActiveMQConnection.doAsyncSendPacket(ActiveMQConnection.java:1266) at org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection.java:1260) at org.apache.activemq.ActiveMQSession.<init>(ActiveMQSession.java:250) at org.apache.activemq.ActiveMQConnection.createSession(ActiveMQConnection.java:319) at sun.reflect.GeneratedMethodAccessor331.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:37) at java.lang.reflect.Method.invoke(Method.java:611) at org.springframework.jms.connection.SingleConnectionFactory$SharedConnectionInvocationHandler.invoke(SingleConnectionFactory.java:550) at $Proxy77.createSession(Unknown Source) at org.springframework.jms.support.JmsAccessor.createSession(JmsAccessor.java:196) at org.springframework.jms.core.JmsTemplate.execute(JmsTemplate.java:457) at org.springframework.jms.core.JmsTemplate.send(JmsTemplate.java:534)
        Hide
        Robert Elliot added a comment -

        It may or may not be relevant, but we also see null pointer exceptions as the queue fills up:
        2011-07-27 19:56:23,509 [ActiveMQ Task-145-13] WARN o.a.a.b.TransportConnection.Service - Async error occurred: java.lang.NullPointerException
        java.lang.NullPointerException: null
        at org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:500) ~[activemq-core-5.5.0.jar:5.5.0]
        at org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:105) ~[activemq-core-5.5.0.jar:5.5.0]
        at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306) ~[activemq-core-5.5.0.jar:5.5.0]
        at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179) [activemq-core-5.5.0.jar:5.5.0]
        at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) [activemq-core-5.5.0.jar:5.5.0]
        at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69) [activemq-core-5.5.0.jar:5.5.0]
        at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:218) [activemq-core-5.5.0.jar:5.5.0]
        at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:127) [activemq-core-5.5.0.jar:5.5.0]
        at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48) [activemq-core-5.5.0.jar:5.5.0]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_17]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_17]
        at java.lang.Thread.run(Thread.java:619) [na:1.6.0_17]

        Show
        Robert Elliot added a comment - It may or may not be relevant, but we also see null pointer exceptions as the queue fills up: 2011-07-27 19:56:23,509 [ActiveMQ Task-145-13] WARN o.a.a.b.TransportConnection.Service - Async error occurred: java.lang.NullPointerException java.lang.NullPointerException: null at org.apache.activemq.broker.TransportConnection.processAddProducer(TransportConnection.java:500) ~ [activemq-core-5.5.0.jar:5.5.0] at org.apache.activemq.command.ProducerInfo.visit(ProducerInfo.java:105) ~ [activemq-core-5.5.0.jar:5.5.0] at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:306) ~ [activemq-core-5.5.0.jar:5.5.0] at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:179) [activemq-core-5.5.0.jar:5.5.0] at org.apache.activemq.transport.ResponseCorrelator.onCommand(ResponseCorrelator.java:116) [activemq-core-5.5.0.jar:5.5.0] at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69) [activemq-core-5.5.0.jar:5.5.0] at org.apache.activemq.transport.vm.VMTransport.iterate(VMTransport.java:218) [activemq-core-5.5.0.jar:5.5.0] at org.apache.activemq.thread.PooledTaskRunner.runTask(PooledTaskRunner.java:127) [activemq-core-5.5.0.jar:5.5.0] at org.apache.activemq.thread.PooledTaskRunner$1.run(PooledTaskRunner.java:48) [activemq-core-5.5.0.jar:5.5.0] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_17] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_17] at java.lang.Thread.run(Thread.java:619) [na:1.6.0_17]
        Hide
        Robert Elliot added a comment -

        Spring JMS configuration

        Show
        Robert Elliot added a comment - Spring JMS configuration
        Hide
        Robert Elliot added a comment -

        Output from JStack - all catalina-exec threads look the same

        Show
        Robert Elliot added a comment - Output from JStack - all catalina-exec threads look the same

          People

          • Assignee:
            Unassigned
            Reporter:
            Robert Elliot
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development