[CAMEL-16790] camel-pulsar is prone to uneven message distribution with large backlog

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.12.0
• Component/s: camel-pulsar

Environment

camel 3.5.0, pulsar 2.7.1, unix

• Estimated Complexity: Unknown
• Assignee: Unassigned
• Reporter: Kai Levy
• Created: 06/Jul/21 22:24
• Updated: 29/Jul/21 13:02
• Resolved: 29/Jul/21 13:02

Description

I have filed a bug in the pulsar project about uneven distribution of messages using the Java client:  https://github.com/apache/pulsar/issues/11008 - there are some reproduction instructions in this ticket. This means that for large backlogs, only the instances that were connected when the messages were produced will be able to process the messages.

The issue occurs when you use a consumer with `messageListener` and set `receiverQueueSize > 0`. It appears that the camel-pulsar component uses that message listener: https://github.com/apache/camel/blob/main/components/camel-pulsar/src/main/java/org/apache/camel/component/pulsar/utils/consumers/CommonCreationStrategyImpl.java#L54

 

We observed the issue in our camel application using camel 3.5.0, but I believe it occurs on more versions. If the issue is not fixed in the java client, a convenient workaround could be to use a threadpool calling `consumer.receive()` in a loop instead of `messageListener`.

Issue Links

• links to: GitHub Pull Request #5872