Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Duplicate
-
5.5.1
-
None
-
None
-
ActiveMQ: 5.5.1-fuse-01-13
JVM: 1.6.0_30-b12
Description
A message appears to have gotten into a stuck state where it cannot be expired from KahaDB.
Each expiry pass (every 30 seconds) is failing, presumably on the same message as the destination/expiration/timestamp/brokerInTime/etc are all identical:
2012-01-23 14:02:48,191 ERROR ranteed.2] Scheduler BrokerLauncher org.apache.activemq.broker.ConnectionContext@7c0e6709: Persistent message has expired: ActiveMQBytesMessage {commandId = 1611, responseRequired = true, messageId = ID:broker1-59818-1327207516246-2:1:1:2:1, originalDestination = null, originalTransactionId = null, producerId = ID:broker-38443-1327100544854-21:1:1:1, destination = topic://TOPIC, transactionId = null, expiration = 1327294033981, timestamp = 1327207633981, arrival = 0, brokerInTime = 1327207633998, brokerOutTime = 1327207633993, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@35fa419, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false} ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null } for DurableTopicSubscription-NC_broker.1_inbound_broker.1.SystemGuaranteed.2:NC-DS_broker.1.SystemGuaranteed.2_TOPIC, id=OFFLINE:1:17, active=false, destinations=1, total=9, pending=9, dispatched=0, inflight=0, prefetchExtension=0
2012-01-23 14:02:48,191 ERROR ranteed.2] Scheduler BrokerLauncher Sent to DLQ: ActiveMQBytesMessage {commandId = 1611, responseRequired = true, messageId = ID:broker1-59818-1327207516246-2:1:1:2:1, originalDestination = null, originalTransactionId = null, producerId = ID:broker-38443-1327100544854-21:1:1:1, destination = topic://TOPIC, transactionId = null, expiration = 1327294033981, timestamp = 1327207633981, arrival = 0, brokerInTime = 1327207633998, brokerOutTime = 1327207633993, correlationId = null, replyTo = null, persistent = true, type = null, priority = 4, groupID = null, groupSequence = 0, targetConsumerId = null, compressed = false, userID = null, content = org.apache.activemq.util.ByteSequence@35fa419, marshalledProperties = null, dataStructure = null, redeliveryCounter = 0, size = 0, properties = null, readOnlyProperties = false, readOnlyBody = false, droppable = false} ActiveMQBytesMessage{ bytesOut = null, dataOut = null, dataIn = null } DurableTopicSubscription-NC_broker.1_inbound_broker.2:NC-DS_broker.2_TOPIC, id=OFFLINE:1:17, active=false, destinations=1, total=9, pending=9, dispatched=0, inflight=0, prefetchExtension=0 org.apache.activemq.broker.ConnectionContext@7c0e6709
2012-01-23 14:02:48,192 ERROR ranteed.2] Scheduler o.a.a.broker.region.Topic Failed to remove expired Message from the store
java.lang.NullPointerException: null
at org.apache.activemq.store.kahadb.MessageDatabase.removeAckLocation(MessageDatabase.java:1920) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1197) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:990) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.kahadb.page.Transaction.execute(Transaction.java:760) ~[kahadb-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:988) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:923) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.data.KahaRemoveMessageCommand.visit(KahaRemoveMessageCommand.java:220) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:915) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:827) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.doAcknowledge(KahaDBStore.java:693) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.acknowledge(KahaDBStore.java:679) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.KahaDBTransactionStore.acknowledge(KahaDBTransactionStore.java:492) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.store.kahadb.KahaDBTransactionStore$2.acknowledge(KahaDBTransactionStore.java:201) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.broker.region.Topic.acknowledge(Topic.java:494) [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.broker.region.Topic.messageExpired(Topic.java:686) [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.broker.region.Topic.doBrowse(Topic.java:568) [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.broker.region.Topic.access$100(Topic.java:63) [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.broker.region.Topic$6.run(Topic.java:667) [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at org.apache.activemq.thread.SchedulerTimerTask.run(SchedulerTimerTask.java:33) [activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13]
at java.util.TimerThread.mainLoop(Unknown Source) [na:1.6.0_30]
at java.util.TimerThread.run(Unknown Source) [na:1.6.0_30]
Restarting the broker does not help. Restarting the client causes the same exception to be serialised to the client:
2012-01-23 01:40:49,903 ERROR true&soTimeout=60000 ErrorListener Error javax.jms.JMSException: java.lang.NullPointerException at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:49) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:1855) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.ActiveMQConnection$2$1.run(ActiveMQConnection.java:1776) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source) [na:1.6.0_30] at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source) [na:1.6.0_30] at java.lang.Thread.run(Unknown Source) [na:1.6.0_30] Caused by: java.lang.NullPointerException: null at org.apache.activemq.store.kahadb.MessageDatabase.removeAckLocation(MessageDatabase.java:1920) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.MessageDatabase.updateIndex(MessageDatabase.java:1197) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.MessageDatabase$15.execute(MessageDatabase.java:990) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.kahadb.page.Transaction.execute(Transaction.java:760) ~[kahadb-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:988) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.MessageDatabase$13.visit(MessageDatabase.java:923) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.data.KahaRemoveMessageCommand.visit(KahaRemoveMessageCommand.java:220) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.MessageDatabase.process(MessageDatabase.java:915) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.MessageDatabase.store(MessageDatabase.java:827) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.doAcknowledge(KahaDBStore.java:693) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.KahaDBStore$KahaDBTopicMessageStore.acknowledge(KahaDBStore.java:679) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.KahaDBTransactionStore.acknowledge(KahaDBTransactionStore.java:492) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.store.kahadb.KahaDBTransactionStore$2.acknowledge(KahaDBTransactionStore.java:201) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.region.Topic.acknowledge(Topic.java:494) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.region.DurableTopicSubscription.acknowledge(DurableTopicSubscription.java:270) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.region.PrefetchSubscription.acknowledge(PrefetchSubscription.java:229) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:427) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:568) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:77) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.BrokerFilter.acknowledge(BrokerFilter.java:77) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:276) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:87) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.MutableBrokerFilter.acknowledge(MutableBrokerFilter.java:87) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.TransportConnection.processMessageAck(TransportConnection.java:477) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.command.MessageAck.visit(MessageAck.java:229) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:318) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:181) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.transport.MutexTransport.onCommand(MutexTransport.java:50) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.transport.AbstractInactivityMonitor.onCommand(AbstractInactivityMonitor.java:229) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.transport.nio.NIOTransport.serviceRead(NIOTransport.java:138) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.transport.nio.NIOTransport$1.onSelect(NIOTransport.java:69) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.transport.nio.SelectorSelection.onSelect(SelectorSelection.java:94) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] at org.apache.activemq.transport.nio.SelectorWorker$1.run(SelectorWorker.java:119) ~[activemq-core-5.5.1-fuse-01-13.jar:5.5.1-fuse-01-13] ... 3 common frames omitted
Is there any mechanism to purge the stuck message (without blowing away the whole database), or deal with the situation automatically?
Attachments
Issue Links
- duplicates
-
AMQ-3634 Automatic recovery on startup when index is corrupted
-
- Resolved
-