Durable consumers recovery

• Key: AMQ-2303
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Dejan Bosanac
• Reporter: Dejan Bosanac
• Component/s: Broker
• Affects Version/s: 5.2.0
• Fix Version/s: 5.3.0

Description

When a broker with a lots of durable consumers goes down and is restarted with forced store recovery, producer fails with the following exception afterwards

2009-06-24 17:54:02,923 [127.0.0.1:55928] ERROR AbstractStoreCursor            - Failed to fill batch
java.lang.IllegalStateException: Message id ID:houdfrazierc.americas.progress.com-55015-1245858822880-1:45:1:1:1 could not be recovered from the data store for: topic://failoverTopic - already dispatched
	at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:62)
	at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:88)
	at org.apache.activemq.store.kahadaptor.KahaTopicReferenceStore.recoverNextMessages(KahaTopicReferenceStore.java:307)
	at org.apache.activemq.store.amq.AMQTopicMessageStore.recoverNextMessages(AMQTopicMessageStore.java:60)
	at org.apache.activemq.broker.region.cursors.TopicStorePrefetch.doFillBatch(TopicStorePrefetch.java:91)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.fillBatch(AbstractStoreCursor.java:227)
	at org.apache.activemq.broker.region.cursors.AbstractStoreCursor.reset(AbstractStoreCursor.java:99)
	at org.apache.activemq.broker.region.cursors.StoreDurableSubscriberCursor.reset(StoreDurableSubscriberCursor.java:224)
	at org.apache.activemq.broker.region.PrefetchSubscription.dispatchPending(PrefetchSubscription.java:574)
	at org.apache.activemq.broker.region.PrefetchSubscription.add(PrefetchSubscription.java:158)
	at org.apache.activemq.broker.region.DurableTopicSubscription.add(DurableTopicSubscription.java:197)
	at org.apache.activemq.broker.region.policy.SimpleDispatchPolicy.dispatch(SimpleDispatchPolicy.java:49)
	at org.apache.activemq.broker.region.Topic.dispatch(Topic.java:577)
	at org.apache.activemq.broker.region.Topic.doMessageSend(Topic.java:430)
	at org.apache.activemq.broker.region.Topic.send(Topic.java:368)
	at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:354)
	at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:443)
	at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
	at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
	at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:133)
	at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:455)
	at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:639)
	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:308)
	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:182)
	at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:68)
	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:113)
	at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:210)
	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:84)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:203)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:185)
	at java.lang.Thread.run(Thread.java:613)

Comments

[Dejan Bosanac]

Resolved by writing a SubscriptionInfo to the journal when durable consumer subscribes and using this info to recover it at the right moment (instead of the beginning of the recovery).