Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-4239

Possible Race Condition in FilePendingMessageCursor when sending messages quickly

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Incomplete
    • Affects Version/s: 5.7.0
    • Fix Version/s: None
    • Component/s: Broker
    • Environment:

      Description

      When sending persistent messages extremely quickly, we often see the following stack trace.

      Broker Persistence is turned on, but not using KahaDB. Instead, using JDBC Persistence (MySQL). KahaDB is commented out on the xml configuration file.

      Reading through source, seems like the Kaha engine is still used for temporary storage of pending messages (/data/**/tmp_storage). However, in reading through code, seems like the "loaded" member variable is not intialized in time, causing this stack to be thrown.

      Any help appreciated, simply gums up our logging, and the exceptions make it back to the client.

      [04 Dec 2012 08:02:08] [scheduler_Worker-2] 36391186 ERROR
      (com.ms.jms.publisher.BaseMessagePublisher:143) -
      java.lang.IllegalStateException: TheListIndex is not loaded
      javax.jms.JMSException: java.lang.IllegalStateException: TheListIndex
      is not loaded
      at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49)
      at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1377)
      at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1305)
      at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1795)
      at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:277)
      at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:212)
      at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:269)
      at org.springframework.jms.connection.CachedMessageProducer.send(CachedMessageProducer.java:121)
      at com.ms.jms.publisher.BaseMessagePublisher.talker(BaseMessagePublisher.java:133)
      at com.ms.jms.publisher.BaseMessagePublisher.sendMessage(BaseMessagePublisher.java:63)
      at com.ms.mscm.bus.ClientBasedMessagePublisher.sendMessage(ClientBasedMessagePublisher.java:169)
      at com.ms.cart.ScheduledCartProcessor.checkout(ScheduledCartProcessor.java:101)
      at com.ms.cart.ScheduledCartJob.doExecuteInternal(ScheduledCartJob.java:63)
      at com.ms.cart.ScheduledCartJob.executeInternal(ScheduledCartJob.java:50)
      at org.springframework.scheduling.quartz.QuartzJobBean.execute(QuartzJobBean.java:86)
      at org.quartz.core.JobRunShell.run(JobRunShell.java:216)
      at org.quartz.simpl.SimpleThreadPool$WorkerThread.run(SimpleThreadPool.java:549)
      Caused by: java.lang.RuntimeException:
      java.lang.IllegalStateException: TheListIndex is not loaded
      at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:239)
      at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.addMessageLast(FilePendingMessageCursor.java:202)
      at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:1669)
      at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:785)
      at org.apache.activemq.broker.region.Queue.send(Queue.java:717)
      at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:407)
      at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:503)
      at org.apache.activemq.broker.jmx.ManagedRegionBroker.send(ManagedRegionBroker.java:311)
      at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
      at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:96)
      at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:317)
      at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:129)
      at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
      at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:450)
      at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:680)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:294)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:152)
      at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
      at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:256)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138)
      at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69)
      at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94)
      at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119)
      at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      at java.lang.Thread.run(Thread.java:662)
      Caused by: java.lang.IllegalStateException: TheListIndex is not loaded
      at org.apache.kahadb.index.ListIndex.assertLoaded(ListIndex.java:301)
      at org.apache.kahadb.index.ListIndex.add(ListIndex.java:199)
      at org.apache.activemq.store.kahadb.plist.PList$2.execute(PList.java:84)
      at org.apache.kahadb.page.Transaction.execute(Transaction.java:769)
      at org.apache.activemq.store.kahadb.plist.PList.addLast(PList.java:82)
      at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.flushToDisk(FilePendingMessageCursor.java:438)
      at org.apache.activemq.broker.region.cursors.FilePendingMessageCursor.tryAddMessageLast(FilePendingMessageCursor.java:226)
      ... 27 more

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              joechang Joseph Chang
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: