Details
-
Bug
-
Status: Closed
-
Critical
-
Resolution: Not A Problem
-
5.4.2, 5.5.0, 5.6.0
-
None
-
None
-
Windows server 2008
12GB Ram
12 cores
spring integration
embedded activemq network of brokers
Description
i'm encountering critical problems with activemq in our dev enviroment,
activemq brokers seem to hang and not respond to other brokers or jms consumers/producers.
my example setup is an activemq network of embedded brokers (2 to be precise),
one process with an embedded broker produces work,
a worker process with an embedded broker as well that does the work.
2 queues, a running-queue and a stopped-queue, each msg has a counter which is decremented every time work is done on it until 0 and then the msg is passed to the stopped-queue, where it is consumed by the work producer.
the logic is implemented with basic spring integration.
activemq is configured with plenty of memory to spare, and no persistency, jmx is exposed to allow monitoring of the queues,
what happens is that occasionally msg's get stuck inflight, meaning they do not get an ack, sometimes the inflight is between brokers, and sometimes between an embedded broker and it's internal jvm consumer.
the worker broker seems to hang, the jmx statistics seem to be frozen, although they do refresh, the broker responds to force restarts of the network connector, and sometime the restart frees msg's until it gets stuck again.
if i leave the worker on after it's stuck and relaunch the producer, both brokers don't seem to communicate, the produced msg's on one side, don't reach the other, same goes for all advisory msg's related stats (such as producer/consumer counts)
i tried changing every bit of configuration, with the broker, and with spring itself, but i have a feeling the problem doesn't have anything to do with spring (because of inner broker communication issues), the problem is easily reproducible on a powerful machine where the msg flow is fast, adding persistency or doing it on a normal desktop slows everything down and makes it more difficult to reproduce,
i tried all recent activemq versions up to the latest 5.6 snapshot.
i attached the relevent junit tests, one needs to launch the agentTest, and then the producerTest,
my recommendation is leaving agentTest on and relaunching producerTest over and over again, producerTest should be finished when all it's messages are done (do not launch 2 producer processes at the same time otherwise they will never be finished)
relevant maven test cases :
http://db.tt/diVWPNnJ