Announcement of new temporary queue may lose race with message pointing to it in network of brokers.

• Key: AMQ-2214
• Type: Bug
• Status: Closed
• Resolution: Working as Designed
• Priority: Major
• Assignee: Unassigned
• Reporter: Jörg Henne
• Component/s: Broker
• Affects Version/s: 5.2.0

Description

Scenario:

• one well-known initiation queue is used by
• clients and servers to
• establish a private session using a pair of temporary queues:
  1. client creates temporary queue
  2. creates message with reply-to pointing to temporary queue
  3. sends message to initiation queue
  4. server receives initiation message
  5. creates temporary queue
  6. replys with handshake reply with reply-to pointing to second temporary queue
• clients repeatedly exercise this chat scenario
• communication is provided over a network of brokers which is established using auto discovery.

Problem:

When this scenario is executed over a network of brokers, sometimes the temporary queue has not yet been established thoughout the network of brokers when one of the partners already received a message with reply-to set to the temporary queue. This leads to exceptions like this:

javax.jms.InvalidDestinationException: Cannot publish to a deleted Destination: temp-queue://ID:jh-xp-2-1250-1240304283337-2:0:378
	at org.apache.activemq.ActiveMQSession.send(ActiveMQSession.java:1597)
	at org.apache.activemq.ActiveMQMessageProducer.send(ActiveMQMessageProducer.java:227)
	at org.apache.activemq.ActiveMQMessageProducerSupport.send(ActiveMQMessageProducerSupport.java:241)
	at Client.chat(Client.java:46)
	at Client.main(Client.java:78)

The problem seems to be heavily timing-dependant as it is very hard to reproduce with several brokers running on the same machine. However, it is more easily reproduced with brokers running in separate virtual machines (VM as in VMWare) or even distributed across real hardware.

Work-around

A work-around for this problem (and the proof that the temporary queue "springs into existence" shortly after the exception is thrown) is depicted in the following retry code:

int trys = 5;
while (true)
	try {
		MessageProducer cp = s.createProducer(rq);
		cp.send(s.createTextMessage(rr));
		break;
	} catch (InvalidDestinationException e) {
		if (--trys > 0) {
			logger.error("Got InvalidDestinationException - retrying " + trys, e);
			Thread.sleep(100);
			continue;
		} else
			throw e;
	}

Comments

[Jörg Henne]

Sample client, server and broker configuration. Watch out for "Got InvalidDestinationException" messages.

[Gary Tully]

I see you linked to the AMQ-1176 - did you consider setting watchTopicAdvisories to false for your connectionFactory, this will ensure the the JMS client will send the message to the broker and create the temp destination on the fly. The origin of the current client exception is that it does not yet know about the temp destination but the broker should at this point.

[Jörg Henne]

No, I didn't. My bad. My excuse for this is that the way I read the comment about watchTopicAdvisories I gathered that this was a work-around which was no longer needed due to the code change and this only applied to topics.

There is no Javadoc for ActiveMQConnectionFactory.watchTopicAdvisories - is there any description of the implications of setting this to false?

Thanks!