Producer block when queue memory limit is setted to 5Mb and no consumer is actived

• Key: AMQ-1868
• Type: Bug
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Alessandro Zanfardino
• Component/s: Broker
• Affects Version/s: 5.1.0
• Fix Version/s: 5.3.0

Description

Hallo
I'm using the attached source code to test AMQ 5.1.0

I'm using following setting:
//SESSION
ackMode = Session.AUTO_ACKNOWLEDGE;
//FACTORY
dispatchAsync = true;
useAsyncSend = false;
//PRODUCERS
DelveryMode=Pesrsistent
timeToLive = 21600000; //6h

If I use the following configuration in activemq.xml:

<memoryUsage limit="20 mb"/>
<policyEntry queue=">" memoryLimit="5mb"/>

I noted this behaviour:

if producer starts a loop of sendMessage on a queue where no consumer is listen, after N messages, when the 5Mb are full, producer is blocked.

Block ends when some consumer registers a listener on this queue and start to conuming messages.

If I change the configuration, removing queue memory limit like below:

<memoryUsage limit="20 mb"/>
<policyEntry queue=">" />

and producer starts a loop of sendMessage on a queue where no consumer is listen, producer is NEVER blocked.

Memory usage percentage go up until 90% and then remains steady, but producer continues to send.

I expected this same behaviour when queue memory limit was 5Mb, also.

Further, I set values for queue memory limit higher than 5Mb and repeated the test :

memoryLimit="10 mb" => producer was blocked

memoryLimit="18 mb" (=90% of 20Mb) => producer was blocked

memoryLimit="19 mb" ( >90% of 20Mb) => producer NEVER blocked

I expected that producer NEVER blocked in every cases.

Please can you help me for this issue?
Feel free to ask me other details.

Comments

[Hans Bausewein]

Other people expect that messages are never lost!

Do you have a solution? An infinite amount of memory?

This describes the issue more or less:

http://activemq.apache.org/slow-consumer-handling.html

It depends very much on your application, what's the best solution.

I'd say, if you need long term storage, use a database.

In many cases you can use the size of the queue as a signal that something is going terribly wrong.
Also it could be that messages become worthless as they get old. (... and "old" could be "seconds").

In a dynamic application a blocked producer is usually the best. As soon as messages can be delivered, it will be unblocked and continue producing more messages.

If producer and consumers are on the same machine or use the same resources it's a nice way to distribute the load in a proper way. A blocked producer does not cost resources, so they are available for the consumers.

Hans

[Hans Bausewein]

Maybe you've found a bug if the producer was not blocked, when the memory limit was reached.
In that case messages would be lost.

I'm not sure whether I completely understand your issue.

[Hans Bausewein]

One more:

http://activemq.apache.org/slow-consumers.html

I think persistent queue messages should be saved to disk, so the memory limit should not be an issue.
Maybe one of the developers can say something better about this?

But in the end you'll probably have the same issue: when the disk limit is reached, your producer will block.

... until a consumer takes some messages.

Hans

[Alessandro Zanfardino]

Hi Hans,
thank you for your answer.

I'm using the default message cursor type: Store based.

http://activemq.apache.org/message-cursors.html

When my Producer starts, broker works like in the picture "Dispatching Messages for Fast Customer".

When Dispatch Queue becomes full, broker should work like in the picture "Dispatching Messages if Dispatch Queue is Full".

It seems that

• If I use no memory limit for queue, or I use memory limit close to Total Memory
  When Dispatch Queue becomes full then messages are paged in to the dispatch queue from a pending cursor on demand,
  and producer is not blocked (limited only from disk sapce)

• If I use 5Mb memory limit for queue,
  When Dispatch Queue becomes full producer is blocked, rather than to swith to model that use Pending Cursor

So, my question is: why if I use 5Mb memory limit for queue, broker doesn't work like when I use no limit (or use high limit)?

Alessandro