Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
5.16.3
-
None
-
None
Description
Summary:
Under heavy loads, I'm observing that some JMS queues start to fill up far beyond the workload being done, causing my application to do needless work. Once I stop the load generators, the high number of messages usually get consumed quickly.
Details:
I am one of the performance QA testers for a Java based order management solution. Orders are created and moved through various statuses in the system (referred to as an order pipeline). Stand alone JVMs (referred to as agents) do a specific task to move orders through the system. A JDBC database stores data and JMS is used for the agents to communicate and divide work.
For example, a simple pipeline would be:
Created -> Scheduled -> Released -> Shipped
There would be 4 agents in this pipeline, one for each of the statuses, and each agent has its own JMS queue. The agents are all multi-threaded. The first thread runs a method called getJobs() which goes to the JDBC database to fetch work. It will fetch a user specified amount of orders (in my case 5,000) and put them into the JMS queue in batches of 5,000. The other threads of the agent run a method called executeJobs() which then, in a multi-threaded fashion, consume these messages and do work. Once the first 5,000 messages are consumed, 5,000 more messages are put in the queue, and so forth. The agents are time triggered, similar to that of a Unix cron job, in that they run every X minutes. All of the agents run asynchronously to each other. Some agents are faster than others (for example, createOrder is faster than scheduleOrder).
An example of the workflow would be:
1.) My tests starts by feeding messages manually into a JMS queue for the createOrder agent to read from - I use Jmeter for this.
2.) The createOrder agent reads these JMS messages and creates orders for them in the JDBC database. These orders are marked that they need to be scheduled.
3.) When the scheduleOrder agent triggers, it goes to the JDBC database and fetches the work it needs by storing it the JMS queue for scheduleOrder. The remaining agent threads process these orders.
4.) When the releaseOrder agent triggers, the same thing happens as it did for scheduleOrder, but this time for orders who need to be released.
5.) When the shipping agent triggers, the same thing again happens.
The Problem:
As a performance QA tester, I try to push the system as much as possible by feeding messages into the createOrder JMS queue as fast as possible and start up multiple agents of each type and change the number of threads of them to tune them. If I push messages into the createOrder queue via Jmeter too fast, createOrder can handle them but scheduleOrder cannot as it has more processing logic. Over time, the scheduleOrder queue starts to fill up much past 5,000, which would be the theoretical maximum of any queue in this configuration. Sometimes it gets into the hundreds of thousands. If I stop the load generator into the createOrder queue, usually after some time this backlog of messages will disappear very fast.
Our application supports other JMS vendors such as Weblogic JMS and IBM MQ which do not see this problem. I am not a developer but have been told that our code is all JMS compliant and there is nothing we can do on our end. I am a big fan of ActiveMQ and am trying to push its adoption by our product but this is a roadblock. This is most likely one or more of the following:
- A bug in ActiveMQ
- Some ActiveMQ config or tuning I need to do
- A bug in our product
I've seen this issue for a few years now and am currently seeing it on ActiveMQ 5.16.3. I've seen this on both RHEL and CentOS 7 and 8. I have not done any ActiveMQ tuning out of the box. Please let me know if I can provide any ActiveMQ logs or debugging analysis and I will gladly share it.