Unable to deliver messages through a mixed version broker network

• Key: AMQ-1472
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Blocker
• Assignee: Rob Davies
• Reporter: Jason Vasquez
• Affects Version/s: 5.0.0
• Fix Version/s: 5.0.0

Description

Setup:

Client Producer:	5.0-SNAPSHOT
Broker A:	4.1.1
Broker B:	5.0-SNAPSHOT
Client Consumer:	5.0-SNAPSHOT

Messages from the producer are sent to Broker A, which discovers Broker B via standard multicast discovery (each broker was given a unique name before starting). Consumer is connected to Broker B, set to consume messages from the same queue.

After a small number of successful transfers, (I've seen between 0 and 7) Broker B will display this message on each send attempt by the producer, and break the network connection between Broker A and B (which is re-established immediately):

ERROR AsyncDataManager               - Looking for key 1 but not found in fileMap: {}
ERROR QueueStorePrefetch             - Failed to fill batch
java.io.IOException: Failed to read to journal for: offset = 172, file = 1, size = -1, type = 0. Reason: java.io.IOException: Could not locate data file data--1
	at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:33)
	at org.apache.activemq.store.amq.AMQPersistenceAdapter.createReadException(AMQPersistenceAdapter.java:558)
	at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:444)
	at org.apache.activemq.store.amq.AMQMessageStore.getMessage(AMQMessageStore.java:383)
	at org.apache.activemq.store.amq.RecoveryListenerAdapter.recoverMessageReference(RecoveryListenerAdapter.java:54)
	at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverReference(KahaReferenceStore.java:68)
	at org.apache.activemq.store.kahadaptor.KahaReferenceStore.recoverNextMessages(KahaReferenceStore.java:98)
	at org.apache.activemq.store.amq.AMQMessageStore.recoverNextMessages(AMQMessageStore.java:495)
	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.fillBatch(QueueStorePrefetch.java:167)
	at org.apache.activemq.broker.region.cursors.QueueStorePrefetch.hasNext(QueueStorePrefetch.java:109)
	at org.apache.activemq.broker.region.cursors.StoreQueueCursor.hasNext(StoreQueueCursor.java:129)
	at org.apache.activemq.broker.region.Queue.buildList(Queue.java:1022)
	at org.apache.activemq.broker.region.Queue.doPageIn(Queue.java:1005)
	at org.apache.activemq.broker.region.Queue.pageInMessages(Queue.java:1068)
	at org.apache.activemq.broker.region.Queue.sendMessage(Queue.java:999)
	at org.apache.activemq.broker.region.Queue.doMessageSend(Queue.java:484)
	at org.apache.activemq.broker.region.Queue.send(Queue.java:441)
	at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:328)
	at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:402)
	at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
	at org.apache.activemq.broker.BrokerFilter.send(BrokerFilter.java:125)
	at org.apache.activemq.broker.CompositeDestinationBroker.send(CompositeDestinationBroker.java:95)
	at org.apache.activemq.broker.MutableBrokerFilter.send(MutableBrokerFilter.java:135)
	at org.apache.activemq.broker.TransportConnection.processMessage(TransportConnection.java:434)
	at org.apache.activemq.command.ActiveMQMessage.visit(ActiveMQMessage.java:623)
	at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281)
	at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
	at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
	at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
	at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:126)
	at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
	at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:183)
	at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:170)
	at java.lang.Thread.run(Thread.java:613)
Caused by: java.io.IOException: Could not locate data file data--1
	at org.apache.activemq.kaha.impl.async.AsyncDataManager.getDataFile(AsyncDataManager.java:285)
	at org.apache.activemq.kaha.impl.async.AsyncDataManager.read(AsyncDataManager.java:505)
	at org.apache.activemq.store.amq.AMQPersistenceAdapter.readCommand(AMQPersistenceAdapter.java:441)
	... 31 more

Seemed like this might be relevant – here's the directory structure for Broker B's data directory – note data file named data-kaha-1 vs. data--1:

$ find data                                                                                                                         3:13PM
data
data/activemq.log
data/localhost50
data/localhost50/journal
data/localhost50/journal/data-control
data/localhost50/kr-store
data/localhost50/kr-store/data
data/localhost50/kr-store/data/data-kaha-1
data/localhost50/kr-store/data/data-queue-data-1
data/localhost50/kr-store/data/hash-index-blob_topic___ActiveMQ.Agent-Subscriptions
data/localhost50/kr-store/data/hash-index-queue-data_queue___DUMMY.QUEUE
data/localhost50/kr-store/data/hash-index-queue-data_queue___example.A
data/localhost50/kr-store/data/hash-index-topic-data_topic___ActiveMQ.Agent
data/localhost50/kr-store/data/index-blob
data/localhost50/kr-store/data/index-kaha
data/localhost50/kr-store/data/index-queue-data
data/localhost50/kr-store/data/index-topic-acks
data/localhost50/kr-store/data/index-topic-data
data/localhost50/kr-store/data/lock
data/localhost50/kr-store/state
data/localhost50/kr-store/state/data-kaha-1
data/localhost50/kr-store/state/data-store-state-1
data/localhost50/kr-store/state/hash-index-store-state_state
data/localhost50/kr-store/state/index-kaha
data/localhost50/kr-store/state/index-store-state
data/localhost50/kr-store/state/index-transactions-state
data/localhost50/kr-store/state/lock
data/localhost50/tmp_storage