We have a local and remote broker connected with a duplex bridge, which is initiated by the remote broker.
Producers are attached to the remote broker, one consumer to the local broker.
The following scenario causes messages to be left in the local store, which are replayed when the local broker is restarted:
- messages are forwarded from the remote broker to the local broker
- messages are dispatched to the local consumer
- the connection between the local and remote broker fails
- the local broker tries to acknowledge the message reception to the remote broker, which fails
- the remote broker reconnects
- the messages are resent
- the local broker correctly identifies them as duplicates, but puts them into the store nevertheless where they remain until the local broker is restarted
- other messages are produced and consumed without a problem
- the local broker is restarted
- the duplicates are now delivered to the local consumer again and of course out of order
This behaviour can be identified by a queue size which does not seem to shrink below a certain number, even if a consumer is connected and consuming other messages.
When the log level is set to TRACE these messages indicate the problem: