ActiveMQ
  1. ActiveMQ
  2. AMQ-3293

ActiveMQ broker hangs after sending many large messages with a TTL, and no DLQ

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 5.4.2
    • Fix Version/s: 5.6.0
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      Description

      While doing some ActiveMQ configuration testing I've run into some interesting where I can consistently make ActiveMQ 5.4.2 hang, unable even to shutdown. In this particular case, I have a producer that repeatedly sends persistent TextMessages containing random contents, of a random length up to 4MB to a single queue, with a 30 second TTL. You'll see in the configuration fragments below that we have sendFailIfNoSpace=true and are using vmQueueCursor with a DLQ policy of processExpired=false.

      At some point, the client will get a ResourceAllocationException, as expected once we hit the store limits. I'm running this client in a loop where it will keep trying, and as messages get expired, it will eventually start sending messages again. This will run for awhile, but then the producer will eventually hang. If I kill and restart the producer, it will continue to hang. The jetty console may show a couple of pending messages in the queue, but any consumer also hangs.

      Attempting to shut down activemq results in an error message that a particular connection "is taking a long time to shutdown", however there are no connections to the indicated port (as that producer was killed).

      You can see full thread dumps, lsof output, etc below.

      In another testcase, I was not using vmQueueCursor but this results in OutOfMemory exceptions repeatedly in ActiveMQ, and attempts to restart ActiveMQ hang right after kahadb journal recovery. I do not have a stacktrace or more info from this particular testcase.

      I can sanitize the producer client testcase if needed and attach it.

      ----------

      Full thread dump Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode):

      "Simple Discovery Agent: java.util.concurrent.ThreadPoolExecutor$Worker@520fa4" daemon prio=1 tid=0x0a079150 nid=0x16c6 in Object.wait() [0x88d73000..0x88d74100]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90bc0818> (a java.lang.Object)
        at org.apache.activemq.transport.discovery.simple.SimpleDiscoveryAgent$1.run(SimpleDiscoveryAgent.java:136)
      • locked <0x90bc0818> (a java.lang.Object)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
        at java.lang.Thread.run(Thread.java:595)

      "NetworkBridge" daemon prio=1 tid=0x0a0819d8 nid=0x16c5 waiting on condition [0x883e0000..0x883e0d80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
      at java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
      at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:595)

      "ActiveMQ Task" daemon prio=1 tid=0x0a337a10 nid=0x16be waiting on condition [0x87cfc000..0x87cfd100]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
      at java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342)
      at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637)
      at org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:939)
      at java.lang.Thread.run(Thread.java:595)

      "Simple Discovery Agent: java.util.concurrent.ThreadPoolExecutor$Worker@1ab90e2" daemon prio=1 tid=0x0a875038 nid=0x1621 in Object.wait() [0x891fc000..0x891fcf80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90bc0818> (a java.lang.Object)
        at org.apache.activemq.transport.discovery.simple.SimpleDiscoveryAgent$1.run(SimpleDiscoveryAgent.java:136)
      • locked <0x90bc0818> (a java.lang.Object)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:651)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:676)
        at java.lang.Thread.run(Thread.java:595)

      "ActiveMQ Task" daemon prio=1 tid=0x09ba8158 nid=0x15ee waiting on condition [0x88059000..0x88059f80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
      at java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342)
      at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637)
      at org.apache.activemq.broker.TransportConnection$3.run(TransportConnection.java:939)
      at java.lang.Thread.run(Thread.java:595)

      "Simple Discovery Agent: java.util.concurrent.ThreadPoolExecutor$Worker@12b6836" daemon prio=1 tid=0x09c814a8 nid=0x15e0 waiting on condition [0x87c7b000..0x87c7bd80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireNanos(AbstractQueuedSynchronizer.java:807)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.tryAcquireNanos(AbstractQueuedSynchronizer.java:1122)
      at java.util.concurrent.SynchronousQueue$Node.waitForPut(SynchronousQueue.java:291)
      at java.util.concurrent.SynchronousQueue.poll(SynchronousQueue.java:443)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:475)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:595)

      "ConcurrentQueueStoreAndDispatch" daemon prio=1 tid=0x09ef2628 nid=0x157f waiting on condition [0x88ff8000..0x88ff8e00]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:595)

      "ActiveMQ Transport: tcp:///10.4.64.66:34904" daemon prio=1 tid=0x09ba7ee8 nid=0x1579 waiting on condition [0x88563000..0x88563e80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
      at java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342)
      at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637)
      at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1599)
      at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:708)
      at org.apache.activemq.broker.region.Queue.send(Queue.java:644)
      at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)
      at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:518)
      at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
      at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
      at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:227)
      at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
      at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
      at org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:192)
      at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
      at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:462)
      at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:677)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:228)

      • locked <0xa08ab6d8> (a org.apache.activemq.transport.InactivityMonitor$1)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
        at java.lang.Thread.run(Thread.java:595)

      "ActiveMQ Connection Dispatcher: /10.4.64.66:34904" daemon prio=1 tid=0x0a1a44d8 nid=0x1578 in Object.wait() [0x8b0fe000..0x8b0fef00]
      at java.lang.Object.wait(Native Method)

      • waiting on <0xa08edc38> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0xa08edc38> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "ActiveMQ Connection Dispatcher: vm://cluster_a#6012" daemon prio=1 tid=0x09fca838 nid=0x2c54 in Object.wait() [0x87fd8000..0x87fd9000]
      at java.lang.Object.wait(Native Method)

      • waiting on <0xa07b6438> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0xa07b6438> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "ActiveMQ Connection Dispatcher: vm://cluster_a#3116" daemon prio=1 tid=0x0a7aeb38 nid=0x6139 in Object.wait() [0x87dfe000..0x87dfef80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0xa0791430> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0xa0791430> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "ActiveMQ Transport: tcp:///10.4.64.66:36292" daemon prio=1 tid=0x0a1a4ba0 nid=0x1b15 waiting on condition [0x890fa000..0x890fb080]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:872)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1179)
      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:201)
      at java.util.concurrent.FutureTask.get(FutureTask.java:80)
      at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:718)
      at org.apache.activemq.broker.region.Queue.send(Queue.java:644)
      at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:365)
      at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:518)
      at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
      at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
      at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:227)
      at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
      at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
      at org.apache.activemq.security.AuthorizationBroker.send(AuthorizationBroker.java:192)
      at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
      at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:462)
      at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:677)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:311)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:185)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:69)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:228)

      • locked <0xa0013660> (a org.apache.activemq.transport.InactivityMonitor$1)
        at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
        at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:220)
        at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:202)
        at java.lang.Thread.run(Thread.java:595)

      "ActiveMQ Connection Dispatcher: /10.4.64.66:36292" daemon prio=1 tid=0x09caed28 nid=0x1b14 in Object.wait() [0x882de000..0x882df100]
      at java.lang.Object.wait(Native Method)

      • waiting on <0xa00133c8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0xa00133c8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "Topic ActiveMQ.Advisory.Consumer.Queue.orchestrationResponse" daemon prio=1 tid=0x09e83730 nid=0xbad in Object.wait() [0x87d7d000..0x87d7e000]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x9167a250> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0x9167a250> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "Topic ActiveMQ.Advisory.Expired.Queue.orchestrationResponse" daemon prio=1 tid=0x09e846b0 nid=0x17b in Object.wait() [0x884e2000..0x884e2d80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x916643d8> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0x916643d8> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "Topic ActiveMQ.Advisory.MessageDLQd.Queue.orchestrationResponse" daemon prio=1 tid=0x0a028150 nid=0x17a in Object.wait() [0x8835f000..0x8835fe00]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x91662a10> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0x91662a10> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "Usage Async Task" daemon prio=1 tid=0x0a0c7e30 nid=0x7fdb waiting on condition [0x880da000..0x880dae80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1841)
      at java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:359)
      at java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:470)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
      at java.lang.Thread.run(Thread.java:595)

      "Topic ActiveMQ.Advisory.Producer.Queue.orchestrationResponse" daemon prio=1 tid=0x0a10d120 nid=0x7fd7 in Object.wait() [0x8815b000..0x8815bf00]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90cc6118> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0x90cc6118> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "Topic ActiveMQ.Advisory.Queue" daemon prio=1 tid=0x0a844bb8 nid=0x7fd6 in Object.wait() [0x881dc000..0x881dcf80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90cc2f48> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0x90cc2f48> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "Queue:orchestrationResponse" daemon prio=1 tid=0x89680bc8 nid=0x7fd5 waiting on condition [0x8825d000..0x8825e000]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:872)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1179)
      at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:201)
      at java.util.concurrent.FutureTask.get(FutureTask.java:80)
      at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAsyncMessage(KahaDBStore.java:337)
      at org.apache.activemq.store.kahadb.KahaDBTransactionStore.removeAsyncMessage(KahaDBTransactionStore.java:457)
      at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.removeAsyncMessage(KahaDBTransactionStore.java:171)
      at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:797)
      at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1527)
      at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1519)
      at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1584)
      at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1574)
      at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1686)
      at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1898)
      at org.apache.activemq.broker.region.Queue.iterate(Queue.java:1425)

      • locked <0x90cb36c8> (a org.apache.activemq.broker.region.Queue$3)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:98)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "Topic ActiveMQ.Advisory.Connection" daemon prio=1 tid=0x89783a98 nid=0x7fd4 in Object.wait() [0x88461000..0x88462080]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90cc5188> (a java.lang.Object)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
      • locked <0x90cc5188> (a java.lang.Object)
        at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

      "Timer-1" prio=1 tid=0x898f7bf8 nid=0x7f9c in Object.wait() [0x885e4000..0x885e4e80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90cf75c8> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
      • locked <0x90cf75c8> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

      "HashSessionScavenger-0" daemon prio=1 tid=0x898834c0 nid=0x7f9b in Object.wait() [0x88665000..0x88665f00]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90d4a800> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
      • locked <0x90d4a800> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

      "qtp14633980-49" prio=1 tid=0x895d3050 nid=0x7f9a waiting on condition [0x886e6000..0x886e6f80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
      at java.lang.Thread.run(Thread.java:595)

      "qtp14633980-48" prio=1 tid=0x896688c0 nid=0x7f99 waiting on condition [0x88767000..0x88768000]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
      at java.lang.Thread.run(Thread.java:595)

      "qtp14633980-47" prio=1 tid=0x896f5660 nid=0x7f98 waiting on condition [0x887e8000..0x887e9080]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
      at java.lang.Thread.run(Thread.java:595)

      "qtp14633980-46" prio=1 tid=0x896f4940 nid=0x7f97 waiting on condition [0x88869000..0x8886a100]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
      at java.lang.Thread.run(Thread.java:595)

      "qtp14633980-45" prio=1 tid=0x89552168 nid=0x7f96 waiting on condition [0x888ea000..0x888ead80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
      at java.lang.Thread.run(Thread.java:595)

      "qtp14633980-44" prio=1 tid=0x89af3680 nid=0x7f95 waiting on condition [0x8896b000..0x8896be00]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
      at java.lang.Thread.run(Thread.java:595)

      "qtp14633980-43" prio=1 tid=0x8afe1d38 nid=0x7f94 waiting on condition [0x88a6d000..0x88a6de80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
      at org.eclipse.jetty.util.BlockingArrayQueue.poll(BlockingArrayQueue.java:320)
      at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:465)
      at java.lang.Thread.run(Thread.java:595)

      "qtp14633980-42 - Acceptor0 SelectChannelConnector@0.0.0.0:7922" prio=1 tid=0x89552d90 nid=0x7f93 runnable [0x889ec000..0x889ecf00]
      at sun.nio.ch.PollArrayWrapper.poll0(Native Method)
      at sun.nio.ch.PollArrayWrapper.poll(PollArrayWrapper.java:100)
      at sun.nio.ch.PollSelectorImpl.doSelect(PollSelectorImpl.java:56)
      at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:69)

      • locked <0x90ca0a98> (a sun.nio.ch.Util$1)
      • locked <0x90ca0a88> (a java.util.Collections$UnmodifiableSet)
      • locked <0x90ca09b0> (a sun.nio.ch.PollSelectorImpl)
        at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:80)
        at org.eclipse.jetty.io.nio.SelectorManager$SelectSet.doSelect(SelectorManager.java:480)
        at org.eclipse.jetty.io.nio.SelectorManager.doSelect(SelectorManager.java:193)
        at org.eclipse.jetty.server.nio.SelectChannelConnector.accept(SelectChannelConnector.java:134)
        at org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:793)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$2.run(QueuedThreadPool.java:436)
        at java.lang.Thread.run(Thread.java:595)

      "ActiveMQ Transport Server: tcp://0.0.0.0:7921?transport.closeAsync=false" daemon prio=1 tid=0x898b32d0 nid=0x7f87 runnable [0x88c71000..0x88c72100]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

      • locked <0x90bac140> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at org.apache.activemq.transport.tcp.TcpTransportServer.run(TcpTransportServer.java:280)
        at java.lang.Thread.run(Thread.java:595)

      "ActiveMQ Transport Server Thread Handler: tcp://0.0.0.0:7921?transport.closeAsync=false" daemon prio=1 tid=0x8966b640 nid=0x7f86 waiting on condition [0x88cf2000..0x88cf2d80]
      at sun.misc.Unsafe.park(Native Method)
      at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:146)
      at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:1879)
      at java.util.concurrent.LinkedBlockingQueue.poll(LinkedBlockingQueue.java:396)
      at org.apache.activemq.transport.tcp.TcpTransportServer$1.run(TcpTransportServer.java:352)
      at java.lang.Thread.run(Thread.java:595)

      "ActiveMQ Data File Writer" daemon prio=1 tid=0x89a21840 nid=0x7f84 in Object.wait() [0x88df4000..0x88df4e80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90ba1318> (a org.apache.kahadb.journal.DataFileAppender$1)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:325)
      • locked <0x90ba1318> (a org.apache.kahadb.journal.DataFileAppender$1)
        at org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:216)

      "ActiveMQ Journal Checkpoint Worker" daemon prio=1 tid=0x898bf0b8 nid=0x7f83 waiting on condition [0x88e75000..0x88e75f00]
      at java.lang.Thread.sleep(Native Method)
      at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:286)

      "KahaDB Scheduler" daemon prio=1 tid=0x8c5d25c0 nid=0x7f82 in Object.wait() [0x88ef6000..0x88ef6f80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90ba0430> (a java.util.TaskQueue)
        at java.util.TimerThread.mainLoop(Timer.java:509)
      • locked <0x90ba0430> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

      "RMI LeaseChecker" daemon prio=1 tid=0x8afe0d38 nid=0x7f7e waiting on condition [0x89079000..0x89079d80]
      at java.lang.Thread.sleep(Native Method)
      at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
      at java.lang.Thread.run(Thread.java:595)

      "RMI RenewClean-[10.4.64.66:39231]" daemon prio=1 tid=0x0a337590 nid=0x7f7c in Object.wait() [0x8917b000..0x8917be80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90b9e178> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
      • locked <0x90b9e178> (a java.lang.ref.ReferenceQueue$Lock)
        at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
        at java.lang.Thread.run(Thread.java:595)

      "GC Daemon" daemon prio=1 tid=0x0a295dc0 nid=0x7f7a in Object.wait() [0x8927d000..0x8927df80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90c48fe8> (a sun.misc.GC$LatencyLock)
        at sun.misc.GC$Daemon.run(GC.java:100)
      • locked <0x90c48fe8> (a sun.misc.GC$LatencyLock)

      "RMI Reaper" prio=1 tid=0x0a295620 nid=0x7f79 in Object.wait() [0x892fe000..0x892ff000]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90ae27e0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
      • locked <0x90ae27e0> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
        at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
        at java.lang.Thread.run(Thread.java:595)

      "RMI TCP Accept-0" daemon prio=1 tid=0x0a295040 nid=0x7f78 runnable [0x894fe000..0x894ff080]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

      • locked <0x90bc0380> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
        at java.lang.Thread.run(Thread.java:595)

      "RMI TCP Accept-7923" daemon prio=1 tid=0x898869b8 nid=0x7f76 runnable [0x8b2e2000..0x8b2e2d80]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

      • locked <0x90b9e450> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
        at java.lang.Thread.run(Thread.java:595)

      "RMI TCP Accept-0" daemon prio=1 tid=0x8be438c8 nid=0x7f6d runnable [0x8bcb7000..0x8bcb7e80]
      at java.net.PlainSocketImpl.socketAccept(Native Method)
      at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

      • locked <0x90b27a40> (a java.net.SocksSocketImpl)
        at java.net.ServerSocket.implAccept(ServerSocket.java:450)
        at java.net.ServerSocket.accept(ServerSocket.java:421)
        at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:31)
        at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
        at java.lang.Thread.run(Thread.java:595)

      "Timer-0" daemon prio=1 tid=0x8be41cc8 nid=0x7f6c in Object.wait() [0x8bd38000..0x8bd38f00]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90b27b28> (a java.util.TaskQueue)
        at java.lang.Object.wait(Object.java:474)
        at java.util.TimerThread.mainLoop(Timer.java:483)
      • locked <0x90b27b28> (a java.util.TaskQueue)
        at java.util.TimerThread.run(Timer.java:462)

      "Low Memory Detector" daemon prio=1 tid=0x8c5cffe0 nid=0x7f6b runnable [0x00000000..0x00000000]

      "CompilerThread1" daemon prio=1 tid=0x8c5cec00 nid=0x7f6a waiting on condition [0x00000000..0x8c079228]

      "CompilerThread0" daemon prio=1 tid=0x8c5cdc80 nid=0x7f69 waiting on condition [0x00000000..0x8c0fa2a8]

      "AdapterThread" daemon prio=1 tid=0x8c5cccc0 nid=0x7f68 waiting on condition [0x00000000..0x00000000]

      "Signal Dispatcher" daemon prio=1 tid=0x8c5cbe88 nid=0x7f67 waiting on condition [0x00000000..0x00000000]

      "Finalizer" daemon prio=1 tid=0x8c5c2840 nid=0x7f66 in Object.wait() [0x8c47d000..0x8c47de00]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90b2a8e8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
      • locked <0x90b2a8e8> (a java.lang.ref.ReferenceQueue$Lock)
        at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
        at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

      "Reference Handler" daemon prio=1 tid=0x8c5c22f8 nid=0x7f65 in Object.wait() [0x8c4fe000..0x8c4fee80]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90b31778> (a java.lang.ref.Reference$Lock)
        at java.lang.Object.wait(Object.java:474)
        at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
      • locked <0x90b31778> (a java.lang.ref.Reference$Lock)

      "main" prio=1 tid=0x09b460c0 nid=0x7f58 in Object.wait() [0xbff23000..0xbff24758]
      at java.lang.Object.wait(Native Method)

      • waiting on <0x90d6bef0> (a [Z)
        at java.lang.Object.wait(Object.java:474)
        at org.apache.activemq.console.command.StartCommand.waitForShutdown(StartCommand.java:161)
      • locked <0x90d6bef0> (a [Z)
        at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:104)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:143)
        at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
        at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:85)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:592)
        at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
        at org.apache.activemq.console.Main.main(Main.java:107)

      "VM Thread" prio=1 tid=0x8c5bff38 nid=0x7f64 runnable

      "GC task thread#0 (ParallelGC)" prio=1 tid=0x09b60ad0 nid=0x7f60 runnable

      "GC task thread#1 (ParallelGC)" prio=1 tid=0x09b61720 nid=0x7f61 runnable

      "GC task thread#2 (ParallelGC)" prio=1 tid=0x09b62358 nid=0x7f62 runnable

      "GC task thread#3 (ParallelGC)" prio=1 tid=0x09b62f90 nid=0x7f63 runnable

      "VM Periodic Task Thread" prio=1 tid=0x8be4cb40 nid=0x7f6e waiting on condition

      ----------

      INFO | The connection to '/10.4.64.66:36292' is taking a long time to shutdown.
      INFO | The connection to '/10.4.64.66:36292' is taking a long time to shutdown.
      INFO | The connection to '/10.4.64.66:36292' is taking a long time to shutdown.
      INFO | The connection to '/10.4.64.66:36292' is taking a long time to shutdown.

      ----------

      java 32600 scott 6u IPv6 19919590 0t0 TCP *:59822 (LISTEN)
      java 32600 scott 9u IPv6 19919615 0t0 TCP *:7923 (LISTEN)
      java 32600 scott 12u IPv6 19919617 0t0 TCP *:39231 (LISTEN)
      java 32600 scott 14u IPv6 20371784 0t0 TCP h1:7922->h2:48935 (ESTABLISHED)
      java 32600 scott 18u IPv6 20371785 0t0 TCP h1:7922->h2:48936 (ESTABLISHED)
      java 32600 scott 20u IPv6 20371786 0t0 TCP h1:7922->h2:48937 (ESTABLISHED)
      java 32600 scott 24u IPv6 20371787 0t0 TCP h1:7922->h2:48938 (ESTABLISHED)
      java 32600 scott 29u IPv6 19919657 0t0 TCP *:7922 (LISTEN)
      java 32600 scott 30u IPv6 20371788 0t0 TCP h1:7922->h2:48939 (ESTABLISHED)
      java 32600 scott 31u IPv6 20371789 0t0 TCP h1:7922->h2:48940 (ESTABLISHED)

      ----------

      <destinationPolicy>
      <policyMap>
      <policyEntries>
      <!-- By default, do not use the DLQ, and use vmcursors -->
      <policyEntry queue=">">
      <deadLetterStrategy>
      <sharedDeadLetterStrategy processExpired="false"/>
      </deadLetterStrategy>
      <pendingQueuePolicy>
      <vmQueueCursor/>
      </pendingQueuePolicy>
      </policyEntry>
      </policyEntries>
      </policyMap>
      </destinationPolicy>
      [...]
      <systemUsage>
      <systemUsage sendFailIfNoSpace="true">
      <memoryUsage>
      <memoryUsage limit="256 mb"/>
      </memoryUsage>
      <storeUsage>
      <storeUsage limit="1 gb" name="foo"/>
      </storeUsage>
      <tempUsage>
      <tempUsage limit="100 mb"/>
      </tempUsage>
      </systemUsage>
      </systemUsage>

      ----------

      rw-rr- 1 scott scott 36621377 2011-04-21 17:35 db-126.log
      rw-rr- 1 scott scott 37281079 2011-04-21 17:35 db-127.log
      rw-rr- 1 scott scott 33556041 2011-04-21 23:17 db-132.log
      rw-rr- 1 scott scott 33030144 2011-04-22 09:25 db-134.log
      rw-rr- 1 scott scott 53248 2011-04-22 09:25 db.data
      rw-rr- 1 scott scott 32824 2011-04-22 09:25 db.redo
      rw-rr- 1 scott scott 0 2011-04-21 17:17 lock

      du -sh on kahadb: 128M

      1. activemq.xml
        4 kB
        Scott Emmons
      2. jms.java
        3 kB
        Scott Emmons

        Issue Links

          Activity

          Hide
          Gary Tully added a comment -

          The test case works on trunk as expected, but yes, the broker does need decent heap resources to deal with spikes.
          In the order of 3*the system usage to be safe for this use case with expiry processing etc.

          Additional hang issue resolved with:
          https://issues.apache.org/jira/browse/AMQ-3357

          Show
          Gary Tully added a comment - The test case works on trunk as expected, but yes, the broker does need decent heap resources to deal with spikes. In the order of 3*the system usage to be safe for this use case with expiry processing etc. Additional hang issue resolved with: https://issues.apache.org/jira/browse/AMQ-3357
          Hide
          Gary Tully added a comment -

          On the OOM, I gave your test case a whirl with trunk, the broker does go steady state at about 200MB but the problem is that it peaks to more than 600MB until GC kicks in.
          The vm stack should be 3*the memory usage limit if we are expecting to max it out in normal operation. More aggressive GC would reduce that.
          The broker does need resources because it pulls messages from a network, keeps them in memory and stores them to disk. There are a bunch of memory allocations that are gc candidates overtime, but there can be large peaks short term as both the marshalled and unmarshalled state are present in memory at some stages.

          The store cursor is best in general. The vmcursor will try and reply messages from the store on restart so if an OOM was the cause of a stop, it will again occur on a restart.
          The default store cursor will keep messages in memory up to its configured limit.

          That hang on start is resolved on trunk, recovery now takes out a write lock from the start. see: https://issues.apache.org/jira/browse/AMQ-3357

          Scott: were there other load related issues that you encountered?
          If you can find them with a 5.6-SNAPSHOT, they will be a priority.

          Show
          Gary Tully added a comment - On the OOM, I gave your test case a whirl with trunk, the broker does go steady state at about 200MB but the problem is that it peaks to more than 600MB until GC kicks in. The vm stack should be 3*the memory usage limit if we are expecting to max it out in normal operation. More aggressive GC would reduce that. The broker does need resources because it pulls messages from a network, keeps them in memory and stores them to disk. There are a bunch of memory allocations that are gc candidates overtime, but there can be large peaks short term as both the marshalled and unmarshalled state are present in memory at some stages. The store cursor is best in general. The vmcursor will try and reply messages from the store on restart so if an OOM was the cause of a stop, it will again occur on a restart. The default store cursor will keep messages in memory up to its configured limit. That hang on start is resolved on trunk, recovery now takes out a write lock from the start. see: https://issues.apache.org/jira/browse/AMQ-3357 Scott: were there other load related issues that you encountered? If you can find them with a 5.6-SNAPSHOT, they will be a priority.
          Hide
          Scott Emmons added a comment -

          Sorry, I don't really know Timothy. At my last job we switched back from ActiveMQ to our previous JMS provider due to issues, and at my current job we'll likely switch to something else soon, as ActiveMQ doesn't do well under high load in large systems. Good luck!

          Show
          Scott Emmons added a comment - Sorry, I don't really know Timothy. At my last job we switched back from ActiveMQ to our previous JMS provider due to issues, and at my current job we'll likely switch to something else soon, as ActiveMQ doesn't do well under high load in large systems. Good luck!
          Hide
          Timothy added a comment -

          Hey Scott! I am facing the same problem, but the difference is that I just send an byte message (around 100MB), and the "OutOfMemoryError" be thrown, the following situation is just like you said. I was thinking is that really a bug? or we have to tune the configuration for AMQ. Because if we really want to send large messages, blob or stream message can be used. Did you try to send blob message?

          Show
          Timothy added a comment - Hey Scott! I am facing the same problem, but the difference is that I just send an byte message (around 100MB), and the "OutOfMemoryError" be thrown, the following situation is just like you said. I was thinking is that really a bug? or we have to tune the configuration for AMQ. Because if we really want to send large messages, blob or stream message can be used. Did you try to send blob message?
          Hide
          Scott Emmons added a comment -

          Thanks, that was a piece we were missing, but it doesn't seem to help - within a few minutes, I was able to hit OOM with a 512MB heap in just a couple of minutes with my test program. Increasing the ActiveMQ heap to 1024MB seems to be OK, but that's not the point - I would expect this case to be stable with a 512MB heap and 256MB limits for memstore and per-destination policy for ">".

          Stack trace:

          Exception in thread "ActiveMQ Broker[cluster_a] Scheduler" java.lang.OutOfMemoryError: Java heap space
          at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228)
          at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:825)
          at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955)
          at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493)
          at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)

          Sorry for the disparate scenarios - we are trying to find one configuration that will result in a reliable and stable configuration...

          Show
          Scott Emmons added a comment - Thanks, that was a piece we were missing, but it doesn't seem to help - within a few minutes, I was able to hit OOM with a 512MB heap in just a couple of minutes with my test program. Increasing the ActiveMQ heap to 1024MB seems to be OK, but that's not the point - I would expect this case to be stable with a 512MB heap and 256MB limits for memstore and per-destination policy for ">". Stack trace: Exception in thread "ActiveMQ Broker [cluster_a] Scheduler" java.lang.OutOfMemoryError: Java heap space at org.apache.activemq.protobuf.BaseMessage.mergeFramed(BaseMessage.java:228) at org.apache.activemq.store.kahadb.MessageDatabase.load(MessageDatabase.java:825) at org.apache.activemq.store.kahadb.KahaDBStore.loadMessage(KahaDBStore.java:955) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$5.execute(KahaDBStore.java:493) at org.apache.kahadb.page.Transaction.execute(Transaction.java:728) Sorry for the disparate scenarios - we are trying to find one configuration that will result in a reliable and stable configuration...
          Hide
          Gary Tully added a comment -

          Hi Scott, lots of info and different scenarios.

          Lets first concentrate on 5.5. You need to configure a destination memory limit to limit the cache for the default store cursor.

          <policyEntry queue=">" memoryLimit="256mb" >

          That will avoid the OOM for persistent messages.

          Show
          Gary Tully added a comment - Hi Scott, lots of info and different scenarios. Lets first concentrate on 5.5. You need to configure a destination memory limit to limit the cache for the default store cursor. <policyEntry queue=">" memoryLimit="256mb" > That will avoid the OOM for persistent messages.
          Hide
          Scott Emmons added a comment -

          Attached configuration - activemq.xml

          Show
          Scott Emmons added a comment - Attached configuration - activemq.xml
          Hide
          Scott Emmons added a comment -

          Attached jms.java - producer testcase. Typically running 5 of these in a loop in parallel. This derived from some JMS sample code found somewhere out on the 'net.

          Show
          Scott Emmons added a comment - Attached jms.java - producer testcase. Typically running 5 of these in a loop in parallel. This derived from some JMS sample code found somewhere out on the 'net.
          Hide
          Scott Emmons added a comment -

          With some additional testing we were also able to hit AMQ-3120.

          We decided to give ActiveMQ 5.5.0 a try, but with the exact same config we were using in 5.4.2 we hit an OutOfMemory within a few minutes. Max heap is 512M, mem store is 256M.

          Exception in thread "ConcurrentQueueStoreAndDispatch" java.lang.OutOfMemoryError: Java heap space
          at org.apache.activemq.util.DataByteArrayOutputStream.ensureEnoughBuffer(DataByteArrayOutputStream.java:249)
          at org.apache.activemq.util.DataByteArrayOutputStream.writeBoolean(DataByteArrayOutputStream.java:140)
          at org.apache.activemq.openwire.v6.BaseDataStreamMarshaller.looseMarshalByteSequence(BaseDataStreamMarshaller.java:628)
          at org.apache.activemq.openwire.v6.MessageMarshaller.looseMarshal(MessageMarshaller.java:296)
          at org.apache.activemq.openwire.v6.ActiveMQMessageMarshaller.looseMarshal(ActiveMQMessageMarshaller.java:111)
          at org.apache.activemq.openwire.v6.ActiveMQTextMessageMarshaller.looseMarshal(ActiveMQTextMessageMarshaller.java:111)
          at org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:166)

          Show
          Scott Emmons added a comment - With some additional testing we were also able to hit AMQ-3120 . We decided to give ActiveMQ 5.5.0 a try, but with the exact same config we were using in 5.4.2 we hit an OutOfMemory within a few minutes. Max heap is 512M, mem store is 256M. Exception in thread "ConcurrentQueueStoreAndDispatch" java.lang.OutOfMemoryError: Java heap space at org.apache.activemq.util.DataByteArrayOutputStream.ensureEnoughBuffer(DataByteArrayOutputStream.java:249) at org.apache.activemq.util.DataByteArrayOutputStream.writeBoolean(DataByteArrayOutputStream.java:140) at org.apache.activemq.openwire.v6.BaseDataStreamMarshaller.looseMarshalByteSequence(BaseDataStreamMarshaller.java:628) at org.apache.activemq.openwire.v6.MessageMarshaller.looseMarshal(MessageMarshaller.java:296) at org.apache.activemq.openwire.v6.ActiveMQMessageMarshaller.looseMarshal(ActiveMQMessageMarshaller.java:111) at org.apache.activemq.openwire.v6.ActiveMQTextMessageMarshaller.looseMarshal(ActiveMQTextMessageMarshaller.java:111) at org.apache.activemq.openwire.OpenWireFormat.marshal(OpenWireFormat.java:166)
          Hide
          Scott Emmons added a comment -

          I mentioned above that in my non-vmQueueCursor testcase "attempts to restart ActiveMQ hang right after kahadb journal recovery" when restarted. This happens with this vmQueueCursor as well. Here's the info:

          $ bin/activemq
          Java Runtime: Sun Microsystems Inc. 1.5.0_22 /opt/java5u22/jre
          Heap sizes: current=4992k free=3984k max=504896k
          JVM args: -Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/home/activemq_cluster/conf; -Dactivemq.home=/home/activemq_cluster -Dactivemq.base=/home/activemq_cluster
          ACTIVEMQ_HOME: /home/activemq_cluster
          ACTIVEMQ_BASE: /home/activemq_cluster
          Loading message broker from: xbean:activemq.xml
          INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@f47bf5: startup date [Fri Apr 22 09:52:12 PDT 2011]; root of context hierarchy
          INFO | PListStore:/home/activemq_cluster/data/cluster_a/tmp_storage started
          INFO | Using Persistence Adapter: KahaDBPersistenceAdapter[/home/activemq_cluster/data/kahadb]
          INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:7923/jmxrmi
          INFO | KahaDB is version 3
          INFO | Recovering from the journal ...
          INFO | Recovery replayed 2 operations from the journal in 0.028 seconds.
          INFO | ActiveMQ 5.4.2 JMS Message Broker (cluster_a) is starting
          INFO | For help or more information please see: http://activemq.apache.org/

          — HANGS HERE —

          kahadb directory has same contents as above, and du -sh shows it still consuming 129M.

          ---------

          Full thread dump Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode):

          "Topic ActiveMQ.Advisory.Expired.Queue.orchestrationResponse" daemon prio=1 tid=0x088978d8 nid=0x2d22 in Object.wait() [0x890f0000..0x890f1080]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x917bf4d0> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
          • locked <0x917bf4d0> (a java.lang.Object)
            at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

          "Topic ActiveMQ.Advisory.MessageDLQd.Queue.orchestrationResponse" daemon prio=1 tid=0x08894428 nid=0x2d21 in Object.wait() [0x89171000..0x89172100]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x917bf5d0> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
          • locked <0x917bf5d0> (a java.lang.Object)
            at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

          "Queue:orchestrationResponse" daemon prio=1 tid=0x08d2fd30 nid=0x2d1f in Object.wait() [0x894f8000..0x894f8d80]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x911c1e58> (a java.lang.Object)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105)
          • locked <0x911c1e58> (a java.lang.Object)
            at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36)

          "ActiveMQ Broker[cluster_a] Scheduler" daemon prio=1 tid=0x092ca540 nid=0x2d1d in Object.wait() [0x891f2000..0x891f2e00]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x91047800> (a java.util.TaskQueue)
            at java.lang.Object.wait(Object.java:474)
            at java.util.TimerThread.mainLoop(Timer.java:483)
          • locked <0x91047800> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

          "ActiveMQ Data File Writer" daemon prio=1 tid=0x092ca8c8 nid=0x2d1c in Object.wait() [0x89273000..0x89273e80]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x91047a90> (a org.apache.kahadb.journal.DataFileAppender$1)
            at java.lang.Object.wait(Object.java:474)
            at org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:325)
          • locked <0x91047a90> (a org.apache.kahadb.journal.DataFileAppender$1)
            at org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:216)

          "ActiveMQ Journal Checkpoint Worker" daemon prio=1 tid=0x089b6500 nid=0x2d1b waiting on condition [0x89f8d000..0x89f8df00]
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
          at java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342)
          at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637)
          at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:690)
          at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:293)

          "KahaDB Scheduler" daemon prio=1 tid=0x092c9410 nid=0x2d1a in Object.wait() [0x8977d000..0x8977df80]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x91047d28> (a java.util.TaskQueue)
            at java.util.TimerThread.mainLoop(Timer.java:509)
          • locked <0x91047d28> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

          "RMI LeaseChecker" daemon prio=1 tid=0x08b66310 nid=0x2d17 waiting on condition [0x893f6000..0x893f7100]
          at java.lang.Thread.sleep(Native Method)
          at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310)
          at java.lang.Thread.run(Thread.java:595)

          "RMI RenewClean-[10.4.64.66:37218]" daemon prio=1 tid=0x08cb7720 nid=0x2d15 in Object.wait() [0x89477000..0x89477e00]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x910d8040> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
          • locked <0x910d8040> (a java.lang.ref.ReferenceQueue$Lock)
            at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501)
            at java.lang.Thread.run(Thread.java:595)

          "GC Daemon" daemon prio=1 tid=0x091aa2a0 nid=0x2d12 in Object.wait() [0x895fa000..0x895faf80]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x91124278> (a sun.misc.GC$LatencyLock)
            at sun.misc.GC$Daemon.run(GC.java:100)
          • locked <0x91124278> (a sun.misc.GC$LatencyLock)

          "RMI Reaper" prio=1 tid=0x08c7f9c0 nid=0x2d11 in Object.wait() [0x8967b000..0x8967c000]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x90b3e2b0> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
          • locked <0x90b3e2b0> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
            at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336)
            at java.lang.Thread.run(Thread.java:595)

          "RMI TCP Accept-0" daemon prio=1 tid=0x092fe4e8 nid=0x2d10 runnable [0x896fc000..0x896fd080]
          at java.net.PlainSocketImpl.socketAccept(Native Method)
          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

          • locked <0x9111bf78> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
            at java.lang.Thread.run(Thread.java:595)

          "RMI TCP Accept-7923" daemon prio=1 tid=0x0907ab00 nid=0x2d0e runnable [0x897fe000..0x897fed80]
          at java.net.PlainSocketImpl.socketAccept(Native Method)
          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

          • locked <0x910d39b0> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
            at java.lang.Thread.run(Thread.java:595)

          "RMI TCP Accept-0" daemon prio=1 tid=0x0866f760 nid=0x2d0c runnable [0x8be34000..0x8be34e80]
          at java.net.PlainSocketImpl.socketAccept(Native Method)
          at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384)

          • locked <0x90c192a8> (a java.net.SocksSocketImpl)
            at java.net.ServerSocket.implAccept(ServerSocket.java:450)
            at java.net.ServerSocket.accept(ServerSocket.java:421)
            at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:31)
            at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340)
            at java.lang.Thread.run(Thread.java:595)

          "Timer-0" daemon prio=1 tid=0x0866d9b0 nid=0x2d0b in Object.wait() [0x8beb5000..0x8beb5f00]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x90c19368> (a java.util.TaskQueue)
            at java.lang.Object.wait(Object.java:474)
            at java.util.TimerThread.mainLoop(Timer.java:483)
          • locked <0x90c19368> (a java.util.TaskQueue)
            at java.util.TimerThread.run(Timer.java:462)

          "Low Memory Detector" daemon prio=1 tid=0x08609be0 nid=0x2d0a runnable [0x00000000..0x00000000]

          "CompilerThread1" daemon prio=1 tid=0x08608800 nid=0x2d09 waiting on condition [0x00000000..0x8c07d228]

          "CompilerThread0" daemon prio=1 tid=0x08607880 nid=0x2d08 waiting on condition [0x00000000..0x8c0fe2a8]

          "AdapterThread" daemon prio=1 tid=0x086068d0 nid=0x2d07 waiting on condition [0x00000000..0x00000000]

          "Signal Dispatcher" daemon prio=1 tid=0x08605990 nid=0x2d06 waiting on condition [0x00000000..0x00000000]

          "Finalizer" daemon prio=1 tid=0x085fb210 nid=0x2d05 in Object.wait() [0x8c5f3000..0x8c5f3e00]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x90bd9468> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120)
          • locked <0x90bd9468> (a java.lang.ref.ReferenceQueue$Lock)
            at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136)
            at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159)

          "Reference Handler" daemon prio=1 tid=0x085fab78 nid=0x2d04 in Object.wait() [0x8c674000..0x8c674e80]
          at java.lang.Object.wait(Native Method)

          • waiting on <0x90bd5ba0> (a java.lang.ref.Reference$Lock)
            at java.lang.Object.wait(Object.java:474)
            at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116)
          • locked <0x90bd5ba0> (a java.lang.ref.Reference$Lock)

          "main" prio=1 tid=0x0851b0c0 nid=0x2cf8 waiting on condition [0xbfb20000..0xbfb222a8]
          at sun.misc.Unsafe.park(Native Method)
          at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746)
          at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076)
          at java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342)
          at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637)
          at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:886)
          at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:833)
          at org.apache.activemq.store.kahadb.data.KahaRemoveMessageCommand.visit(KahaRemoveMessageCommand.java:220)
          at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:825)
          at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:751)
          at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeMessage(KahaDBStore.java:378)
          at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAsyncMessage(KahaDBStore.java:350)
          at org.apache.activemq.store.kahadb.KahaDBTransactionStore.removeAsyncMessage(KahaDBTransactionStore.java:457)
          at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.removeAsyncMessage(KahaDBTransactionStore.java:171)
          at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:797)
          at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1527)
          at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1519)
          at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1584)
          at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1574)
          at org.apache.activemq.broker.region.Queue$5.recoverMessage(Queue.java:283)
          at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:472)
          at org.apache.kahadb.page.Transaction.execute(Transaction.java:728)
          at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recover(KahaDBStore.java:464)
          at org.apache.activemq.store.ProxyMessageStore.recover(ProxyMessageStore.java:52)
          at org.apache.activemq.broker.region.Queue.initialize(Queue.java:269)
          at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:86)
          at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:473)
          at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56)
          at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:123)

          • locked <0x91155e68> (a java.lang.Object)
            at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:298)
            at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
            at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:157)
            at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
            at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
            at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
            at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145)
            at org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81)
            at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
            at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:95)
            at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:198)
            at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:106)
            at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
            at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
            at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:109)
            at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
            at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157)
            at org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1781)
            at org.apache.activemq.broker.BrokerService.start(BrokerService.java:489)
            at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:592)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1536)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1477)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1409)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519)
            at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456)
            at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291)
            at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
          • locked <0x90bd3550> (a java.util.concurrent.ConcurrentHashMap)
            at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288)
            at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
            at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:574)
          • locked <0x90bd3858> (a java.util.concurrent.ConcurrentHashMap)
            at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895)
            at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425)
          • locked <0x90bd7120> (a java.lang.Object)
            at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64)
            at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52)
            at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:117)
            at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:115)
            at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:71)
            at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71)
            at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54)
            at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115)
            at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74)
            at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
            at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:143)
            at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57)
            at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:85)
            at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
            at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
            at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
            at java.lang.reflect.Method.invoke(Method.java:592)
            at org.apache.activemq.console.Main.runTaskClass(Main.java:251)
            at org.apache.activemq.console.Main.main(Main.java:107)

          "VM Thread" prio=1 tid=0x085f86f0 nid=0x2d03 runnable

          "GC task thread#0 (ParallelGC)" prio=1 tid=0x08535ad0 nid=0x2cff runnable

          "GC task thread#1 (ParallelGC)" prio=1 tid=0x08536720 nid=0x2d00 runnable

          "GC task thread#2 (ParallelGC)" prio=1 tid=0x08537358 nid=0x2d01 runnable

          "GC task thread#3 (ParallelGC)" prio=1 tid=0x08537f90 nid=0x2d02 runnable

          "VM Periodic Task Thread" prio=1 tid=0x0852cfe8 nid=0x2d0d waiting on condition

          Show
          Scott Emmons added a comment - I mentioned above that in my non-vmQueueCursor testcase "attempts to restart ActiveMQ hang right after kahadb journal recovery" when restarted. This happens with this vmQueueCursor as well. Here's the info: $ bin/activemq Java Runtime: Sun Microsystems Inc. 1.5.0_22 /opt/java5u22/jre Heap sizes: current=4992k free=3984k max=504896k JVM args: -Xmx512M -Dorg.apache.activemq.UseDedicatedTaskRunner=true -Djava.util.logging.config.file=logging.properties -Dcom.sun.management.jmxremote -Dactivemq.classpath=/home/activemq_cluster/conf; -Dactivemq.home=/home/activemq_cluster -Dactivemq.base=/home/activemq_cluster ACTIVEMQ_HOME: /home/activemq_cluster ACTIVEMQ_BASE: /home/activemq_cluster Loading message broker from: xbean:activemq.xml INFO | Refreshing org.apache.activemq.xbean.XBeanBrokerFactory$1@f47bf5: startup date [Fri Apr 22 09:52:12 PDT 2011] ; root of context hierarchy INFO | PListStore:/home/activemq_cluster/data/cluster_a/tmp_storage started INFO | Using Persistence Adapter: KahaDBPersistenceAdapter [/home/activemq_cluster/data/kahadb] INFO | JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:7923/jmxrmi INFO | KahaDB is version 3 INFO | Recovering from the journal ... INFO | Recovery replayed 2 operations from the journal in 0.028 seconds. INFO | ActiveMQ 5.4.2 JMS Message Broker (cluster_a) is starting INFO | For help or more information please see: http://activemq.apache.org/ — HANGS HERE — kahadb directory has same contents as above, and du -sh shows it still consuming 129M. --------- Full thread dump Java HotSpot(TM) Server VM (1.5.0_22-b03 mixed mode): "Topic ActiveMQ.Advisory.Expired.Queue.orchestrationResponse" daemon prio=1 tid=0x088978d8 nid=0x2d22 in Object.wait() [0x890f0000..0x890f1080] at java.lang.Object.wait(Native Method) waiting on <0x917bf4d0> (a java.lang.Object) at java.lang.Object.wait(Object.java:474) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105) locked <0x917bf4d0> (a java.lang.Object) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) "Topic ActiveMQ.Advisory.MessageDLQd.Queue.orchestrationResponse" daemon prio=1 tid=0x08894428 nid=0x2d21 in Object.wait() [0x89171000..0x89172100] at java.lang.Object.wait(Native Method) waiting on <0x917bf5d0> (a java.lang.Object) at java.lang.Object.wait(Object.java:474) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105) locked <0x917bf5d0> (a java.lang.Object) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) "Queue:orchestrationResponse" daemon prio=1 tid=0x08d2fd30 nid=0x2d1f in Object.wait() [0x894f8000..0x894f8d80] at java.lang.Object.wait(Native Method) waiting on <0x911c1e58> (a java.lang.Object) at java.lang.Object.wait(Object.java:474) at org.apache.activemq.thread.DedicatedTaskRunner.runTask(DedicatedTaskRunner.java:105) locked <0x911c1e58> (a java.lang.Object) at org.apache.activemq.thread.DedicatedTaskRunner$1.run(DedicatedTaskRunner.java:36) "ActiveMQ Broker [cluster_a] Scheduler" daemon prio=1 tid=0x092ca540 nid=0x2d1d in Object.wait() [0x891f2000..0x891f2e00] at java.lang.Object.wait(Native Method) waiting on <0x91047800> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:474) at java.util.TimerThread.mainLoop(Timer.java:483) locked <0x91047800> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "ActiveMQ Data File Writer" daemon prio=1 tid=0x092ca8c8 nid=0x2d1c in Object.wait() [0x89273000..0x89273e80] at java.lang.Object.wait(Native Method) waiting on <0x91047a90> (a org.apache.kahadb.journal.DataFileAppender$1) at java.lang.Object.wait(Object.java:474) at org.apache.kahadb.journal.DataFileAppender.processQueue(DataFileAppender.java:325) locked <0x91047a90> (a org.apache.kahadb.journal.DataFileAppender$1) at org.apache.kahadb.journal.DataFileAppender$2.run(DataFileAppender.java:216) "ActiveMQ Journal Checkpoint Worker" daemon prio=1 tid=0x089b6500 nid=0x2d1b waiting on condition [0x89f8d000..0x89f8df00] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076) at java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637) at org.apache.activemq.store.kahadb.MessageDatabase.checkpointCleanup(MessageDatabase.java:690) at org.apache.activemq.store.kahadb.MessageDatabase$3.run(MessageDatabase.java:293) "KahaDB Scheduler" daemon prio=1 tid=0x092c9410 nid=0x2d1a in Object.wait() [0x8977d000..0x8977df80] at java.lang.Object.wait(Native Method) waiting on <0x91047d28> (a java.util.TaskQueue) at java.util.TimerThread.mainLoop(Timer.java:509) locked <0x91047d28> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "RMI LeaseChecker" daemon prio=1 tid=0x08b66310 nid=0x2d17 waiting on condition [0x893f6000..0x893f7100] at java.lang.Thread.sleep(Native Method) at sun.rmi.transport.DGCImpl$LeaseChecker.run(DGCImpl.java:310) at java.lang.Thread.run(Thread.java:595) "RMI RenewClean- [10.4.64.66:37218] " daemon prio=1 tid=0x08cb7720 nid=0x2d15 in Object.wait() [0x89477000..0x89477e00] at java.lang.Object.wait(Native Method) waiting on <0x910d8040> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) locked <0x910d8040> (a java.lang.ref.ReferenceQueue$Lock) at sun.rmi.transport.DGCClient$EndpointEntry$RenewCleanThread.run(DGCClient.java:501) at java.lang.Thread.run(Thread.java:595) "GC Daemon" daemon prio=1 tid=0x091aa2a0 nid=0x2d12 in Object.wait() [0x895fa000..0x895faf80] at java.lang.Object.wait(Native Method) waiting on <0x91124278> (a sun.misc.GC$LatencyLock) at sun.misc.GC$Daemon.run(GC.java:100) locked <0x91124278> (a sun.misc.GC$LatencyLock) "RMI Reaper" prio=1 tid=0x08c7f9c0 nid=0x2d11 in Object.wait() [0x8967b000..0x8967c000] at java.lang.Object.wait(Native Method) waiting on <0x90b3e2b0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) locked <0x90b3e2b0> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) at sun.rmi.transport.ObjectTable$Reaper.run(ObjectTable.java:336) at java.lang.Thread.run(Thread.java:595) "RMI TCP Accept-0" daemon prio=1 tid=0x092fe4e8 nid=0x2d10 runnable [0x896fc000..0x896fd080] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) locked <0x9111bf78> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340) at java.lang.Thread.run(Thread.java:595) "RMI TCP Accept-7923" daemon prio=1 tid=0x0907ab00 nid=0x2d0e runnable [0x897fe000..0x897fed80] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) locked <0x910d39b0> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340) at java.lang.Thread.run(Thread.java:595) "RMI TCP Accept-0" daemon prio=1 tid=0x0866f760 nid=0x2d0c runnable [0x8be34000..0x8be34e80] at java.net.PlainSocketImpl.socketAccept(Native Method) at java.net.PlainSocketImpl.accept(PlainSocketImpl.java:384) locked <0x90c192a8> (a java.net.SocksSocketImpl) at java.net.ServerSocket.implAccept(ServerSocket.java:450) at java.net.ServerSocket.accept(ServerSocket.java:421) at sun.management.jmxremote.LocalRMIServerSocketFactory$1.accept(LocalRMIServerSocketFactory.java:31) at sun.rmi.transport.tcp.TCPTransport.run(TCPTransport.java:340) at java.lang.Thread.run(Thread.java:595) "Timer-0" daemon prio=1 tid=0x0866d9b0 nid=0x2d0b in Object.wait() [0x8beb5000..0x8beb5f00] at java.lang.Object.wait(Native Method) waiting on <0x90c19368> (a java.util.TaskQueue) at java.lang.Object.wait(Object.java:474) at java.util.TimerThread.mainLoop(Timer.java:483) locked <0x90c19368> (a java.util.TaskQueue) at java.util.TimerThread.run(Timer.java:462) "Low Memory Detector" daemon prio=1 tid=0x08609be0 nid=0x2d0a runnable [0x00000000..0x00000000] "CompilerThread1" daemon prio=1 tid=0x08608800 nid=0x2d09 waiting on condition [0x00000000..0x8c07d228] "CompilerThread0" daemon prio=1 tid=0x08607880 nid=0x2d08 waiting on condition [0x00000000..0x8c0fe2a8] "AdapterThread" daemon prio=1 tid=0x086068d0 nid=0x2d07 waiting on condition [0x00000000..0x00000000] "Signal Dispatcher" daemon prio=1 tid=0x08605990 nid=0x2d06 waiting on condition [0x00000000..0x00000000] "Finalizer" daemon prio=1 tid=0x085fb210 nid=0x2d05 in Object.wait() [0x8c5f3000..0x8c5f3e00] at java.lang.Object.wait(Native Method) waiting on <0x90bd9468> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:120) locked <0x90bd9468> (a java.lang.ref.ReferenceQueue$Lock) at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:136) at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:159) "Reference Handler" daemon prio=1 tid=0x085fab78 nid=0x2d04 in Object.wait() [0x8c674000..0x8c674e80] at java.lang.Object.wait(Native Method) waiting on <0x90bd5ba0> (a java.lang.ref.Reference$Lock) at java.lang.Object.wait(Object.java:474) at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:116) locked <0x90bd5ba0> (a java.lang.ref.Reference$Lock) "main" prio=1 tid=0x0851b0c0 nid=0x2cf8 waiting on condition [0xbfb20000..0xbfb222a8] at sun.misc.Unsafe.park(Native Method) at java.util.concurrent.locks.LockSupport.park(LockSupport.java:118) at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:716) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:746) at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1076) at java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync.wlock(ReentrantReadWriteLock.java:342) at java.util.concurrent.locks.ReentrantReadWriteLock$WriteLock.lock(ReentrantReadWriteLock.java:637) at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:886) at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:833) at org.apache.activemq.store.kahadb.data.KahaRemoveMessageCommand.visit(KahaRemoveMessageCommand.java:220) at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:825) at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:751) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeMessage(KahaDBStore.java:378) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.removeAsyncMessage(KahaDBStore.java:350) at org.apache.activemq.store.kahadb.KahaDBTransactionStore.removeAsyncMessage(KahaDBTransactionStore.java:457) at org.apache.activemq.store.kahadb.KahaDBTransactionStore$1.removeAsyncMessage(KahaDBTransactionStore.java:171) at org.apache.activemq.broker.region.Queue.acknowledge(Queue.java:797) at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1527) at org.apache.activemq.broker.region.Queue.removeMessage(Queue.java:1519) at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1584) at org.apache.activemq.broker.region.Queue.messageExpired(Queue.java:1574) at org.apache.activemq.broker.region.Queue$5.recoverMessage(Queue.java:283) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore$4.execute(KahaDBStore.java:472) at org.apache.kahadb.page.Transaction.execute(Transaction.java:728) at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBMessageStore.recover(KahaDBStore.java:464) at org.apache.activemq.store.ProxyMessageStore.recover(ProxyMessageStore.java:52) at org.apache.activemq.broker.region.Queue.initialize(Queue.java:269) at org.apache.activemq.broker.region.DestinationFactoryImpl.createDestination(DestinationFactoryImpl.java:86) at org.apache.activemq.broker.region.AbstractRegion.createDestination(AbstractRegion.java:473) at org.apache.activemq.broker.jmx.ManagedQueueRegion.createDestination(ManagedQueueRegion.java:56) at org.apache.activemq.broker.region.AbstractRegion.addDestination(AbstractRegion.java:123) locked <0x91155e68> (a java.lang.Object) at org.apache.activemq.broker.region.RegionBroker.addDestination(RegionBroker.java:298) at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:157) at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:145) at org.apache.activemq.security.AuthorizationBroker.addDestination(AuthorizationBroker.java:81) at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151) at org.apache.activemq.broker.region.AbstractRegion.start(AbstractRegion.java:95) at org.apache.activemq.broker.region.RegionBroker.start(RegionBroker.java:198) at org.apache.activemq.broker.jmx.ManagedRegionBroker.start(ManagedRegionBroker.java:106) at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:109) at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:157) at org.apache.activemq.broker.BrokerService$3.start(BrokerService.java:1781) at org.apache.activemq.broker.BrokerService.start(BrokerService.java:489) at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:60) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeCustomInitMethod(AbstractAutowireCapableBeanFactory.java:1536) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1477) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1409) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:519) at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:456) at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:291) at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222) locked <0x90bd3550> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:288) at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190) at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:574) locked <0x90bd3858> (a java.util.concurrent.ConcurrentHashMap) at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:895) at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:425) locked <0x90bd7120> (a java.lang.Object) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:64) at org.apache.xbean.spring.context.ResourceXmlApplicationContext.<init>(ResourceXmlApplicationContext.java:52) at org.apache.activemq.xbean.XBeanBrokerFactory$1.<init>(XBeanBrokerFactory.java:117) at org.apache.activemq.xbean.XBeanBrokerFactory.createApplicationContext(XBeanBrokerFactory.java:115) at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:71) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:71) at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:54) at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:115) at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:74) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:143) at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:57) at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:85) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:592) at org.apache.activemq.console.Main.runTaskClass(Main.java:251) at org.apache.activemq.console.Main.main(Main.java:107) "VM Thread" prio=1 tid=0x085f86f0 nid=0x2d03 runnable "GC task thread#0 (ParallelGC)" prio=1 tid=0x08535ad0 nid=0x2cff runnable "GC task thread#1 (ParallelGC)" prio=1 tid=0x08536720 nid=0x2d00 runnable "GC task thread#2 (ParallelGC)" prio=1 tid=0x08537358 nid=0x2d01 runnable "GC task thread#3 (ParallelGC)" prio=1 tid=0x08537f90 nid=0x2d02 runnable "VM Periodic Task Thread" prio=1 tid=0x0852cfe8 nid=0x2d0d waiting on condition

            People

            • Assignee:
              Gary Tully
              Reporter:
              Scott Emmons
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development