Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Cannot Reproduce
-
2.18.0
-
None
-
AMQP, Broker, JMX, Web Console
-
None
-
PROD
Description
Hello,
Recently we have been facing an issue with amq brokers where the consumer count that shows up in AMQ console for each queue is much higher than what is configured. It is coming as a different number for each queue. We believe that consumer count should always remain 2 across all of the queues since we have 2 qpid dispatch routers acting as consumers. Lately we have observed the consumer count is not consistent across all brokers. However there are no recent changes made to the application. It is all of the sudden acting up.
Below is our architecture.
Microservice(kube) --> LB ---> QPID ---> Brokers(master/slave)
The queues that are having more than 2 consumers looks like they are holding up multiple sessions from each qpid dispatch router for an example: for one queue having 6 consumers there are 6 session ID's. all those 6 id's belong to those qpid's. 1-qpid has two consumers sessions while 2-qpid is having 4 session. Just like that we are seeing numbers like 6,5,3 for other queues.
As part of troubleshooting this issue I found a queue under addresses called $.artemis.internal.sf.artemis-cluster-prod.337d5748-6c39-11eb-9983-0ae398a5a5b7 only on one of the broker. I don't recognize this queue nor we created it. Looks the system somehow created but not sure why. All of our queues are created on qpid, we are not creating any queues on brokers. Routing-type on that queue is coming as MULTICAST
Would like to know if anyone knows how/where to fix these issue.
Versions:
activemq: 2.18
qpid-dispatch router: 1.18