[DISPATCH-2353] More than configured consumers count

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.16.0
• Component/s: Router Node, Routing Engine

Environment

PROD

• Assignee: Unassigned
• Reporter: Ekta
• Created: 10/Jan/23 20:56
• Updated: 10/Jan/23 20:57

Description

Hi There,
 
We have few issues with the qpid-dispatch router where we are seeing multiple consumers are getting created randomly by our qpid dispatch routers. We are unable to trace nor able to replicate this issue in our lower env's. Below is the architecture diagram. Please provide any suggestions or feedback to help us understand this issue better. Thanks 
 
Flow: 
Microservice (Kubernetes) ---> NLB (Load balancer) ---> 2 Qpid-Dispatch Routers acting up as consumers in front of brokers (1.16 version) ---> 4 Broker pairs of activemq(2.18 version) masters and slaves (independent pairs) 
 
Problem statement # 1 
We are seeing more than the configured number of consumers count in our activemq hawtio console causing messages to sit in delivering count which are Un browsable since those messages are currently being delivered to its consumers. Having only two qpid dispatch routers(acting as consumers) infront of our activemq brokers, the count should always remain 2 but at times it is going 3 sometimes 4. To resolve this issue, we are having to bounce the qpid’s to release the stuck/bad consumer so that messages can be processed/consumed. Restarting of the client(microservice) and broker does NOT help in this case. Only component that helps is QPID. 
 
Problem statement # 2
At times we see messages going to delivering count even when there are only two configured consumers(2qpids) showing in activemq hawtio console. We don't know the cause to why the messages get stuck in delivering count. To resolve this, we tried restarting the consumer service but that did not help. Next up we tried restarting the brokers, that did not help and noticed that all the stuck delivering messages, the broker slowly replayed back and Eventually the stuck messages came back to delivering count post broker restart. To resolve this issue, we are having to bounce the qpid’s to release the messages and that fixes the issue. 
 
Version of qpid-dispatch router
qpid-dispatch-router-1.16.0-1.el7.x86_64.rpm
qpid-dispatch-tools-1.16.0-1.el7.noarch.rpm

 

As always any suggestions would be greatly appreciated. Thanks