Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
qpid-cpp-1.37.0
-
None
-
None
Description
When routing data with the use of qpid queue routes, I've noticed situations where the memory starts to rise controlled when the queue on the source broker fills up and the route cannot seem to keep up with the data flow.
The qpidd process eventually consumers all the memory on the source broker and is terminated by the kernel oom-killer
The problem seems to happen when a high rate of data is sent through a qpid queue route from a headers exchange.
The queue starts to fill up as it selects incoming data and once it is full the memory continues to rise as long as there is data flowing through the exchange to the route queue.
Stopping this data and letting the queue 'drain down' through the destination broker instance stops the memory rise but does not deallocate any memory.
I create a push route from source to destination broker with a 1GB source queue with limit ring policy that subscribes to all traffic from the default headers exchange (amq.match) and send data until the source queue fills up
For the sending of data, I have a single qpid-send instace sending to a "xmatch: all" bindings on the headers exhange to a queue route on another broker instance. The queue always manages to keep up for a few minutes then inevitably starts to fill up.
After this point the memory continues to rise up to the point of exhaustion. Stopping the qpid-send at any point stops the memory rise but does not de-allocate any blocks.
From the metrics on the exchange it looks like the issue seems to occur when the route is struggling to keep up with data the source queue is binding to on the exchange.
It is not known what is going on 'internally' when the queue route cannot process an incoming message in time but I would have assumed that the message would be discarded
and any memory used released but that is not what looks to be happening.
The following steps, executed sequentially on the source broker produces the problem (on the dest broker just need to start it up, no queues are needed)
# Start qpid broker
/usr/bin/qpidd --auth=no
# Create queue and push route
qpid-config add queue queuerouteall --max-queue-size=1000000000 --limit-policy=ring -b <source broker address>
qpid-config bind amq.match queuerouteall qrakey all x-match=all -b <source broker address>
qpid-route -s queue add <destination broker address> <source broker address> amq.match queuerouteall
# Send high rate of data to source broker
qpid-send -b <source broker address> -a amq.match -m 0 --send-rate 5000 --content-size 1000000 --connection-options {reconnect:true}
I have also run this scenario on qpid-cpp-server and qpid-tools versions 0.30 and 1.36 and in each case the process consumes all the memory on the server(even though the only queue had a ring limit of 1GB)