ActiveMQ hangs, cannot send message/create temp queue

• Key: AMQ-1375
• Type: Bug
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Trevor Higgins
• Component/s: Broker
• Affects Version/s: 4.1.1
• Fix Version/s: 5.4.0

Description

We are experiencing severe issues with AMQ blocking sending. We have a basic setup with one standalone server process and many clients. Each client has its own temp queue for P2P request/response type communication. There is also a number of topics broadcast to all clients. Our clients receive messages in an MDP fashion with each message being processed on a single threaded executor by an anonymous Runnable as follows:

public void onMessage(final Message message) {
	threadRunner.submit(new Runnable() {		
		public void run() {
			try {				
				for(final IReceiverListener listener : listeners) {
					listener.onReceive(message);
				}
			} catch (Throwable e) {
				e.printStackTrace();
			}
		}
	});
}

Every so often, the entire AMQ blocks and we cannot send anything anymore. We also cannot create any new temp queues, which means no new client can start up (each creates its own temp queue during startup). Connections are be created and from that a session, but the session.createTemporaryQueue() call alwaysy hangs. This is similar to AMQ-1278, but as we are cannot use 5.0 until stable release.......

Here is the stack of the client trying to create a temp queue
at Object.wait(long, int) line: not available [native method]
at CondVar(Object).wait() line: 199
at CondVar.await() line: 75
at ArrayBlockingQueue.take() line: 318
at FutureResponse.getResult() line: 42
at ResponseCorrelator.request(Object) line: 75
at ActiveMQConnection.syncSendPacket(Command) line: 1175
at ActiveMQConnection.createTempDestination(boolean) line: 1594
at ActiveMQSession.createTemporaryQueue() line: 1134
at TempDestinationResolver.resolveDestinationName(Session, String, boolean) line: 16
at SimpleMessageListenerContainer(JmsDestinationAccessor).resolveDestinationName(Session, String) line: 100
at SimpleMessageListenerContainer.createListenerConsumer(Session) line: 188
at SimpleMessageListenerContainer.doInitialize() line: 171
at SimpleMessageListenerContainer(AbstractJmsListeningContainer).initialize() line: 160
at SimpleMessageListenerContainer(AbstractJmsListeningContainer).afterPropertiesSet() line: 133
at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1202
at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1172
at DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 428
at AbstractBeanFactory$1.getObject() line: 251
at DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 156
at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String, Class, Object[]) line: 248
at DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 160
at DefaultListableBeanFactory.preInstantiateSingletons() line: 287
at FileSystemXmlApplicationContext(AbstractApplicationContext).refresh() line: 352
at FileSystemXmlApplicationContext.<init>(String[], ApplicationContext) line: 93
at FileSystemXmlApplicationContext.<init>(String[]) line: 77
at FileSystemXmlApplicationContext.<init>(String) line: 67
at BindingService.<init>() line: 65
at BindingService.getInstance() line: 43
at TopologyServiceFactory.initialize() line: 123
at TopologyManager.initialize() line: 31
at TopologyView$3.run(IProgressMonitor) line: 159
at Worker.run() line: 55

I have attached a JStack of the server process with embedded broker also.

From our client side the only thing I am dubious about is the retention of the reference to the message until the anonymous Runnable is executed. Could this cause issues is for some reason the execution blocked, will the messages build up in AMQ?

From looking at JConsole for the broker, the memoryPercentageUsed is zero so it doesnt appear to be a RAM issue.