[CAMEL-19487] camel-core / camel-bean - Dynamic router instances pass messages to wrong endpoints with @DynamicRouter annotation

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 4.0-M3
• Fix Version/s: 3.20.7, 3.21.1, 3.22.0, 4.0.0
• Component/s: camel-core, tests
• Labels: help-wanted
• Estimated Complexity: Moderate
• Assignee: Claus Ibsen
• Reporter: Peter Palaga
• Created: 22/Jun/23 09:34
• Updated: 24/Oct/23 07:27
• Resolved: 02/Aug/23 15:59

Description

This is actually reopening CAMEL-9311, because DynamicRouterConcurrentPOJOTest that was added to reproduce it is still failing.

First I thought the test is just flaky due to its bad design: it was calling assertMockEndpointsSatisfied() without any synchronizations of the participating threads.

But fixing that by adding

sendToSedaA.join(10000);
sendToSedaB.join(10000);

and replacing assertMockEndpointsSatisfied() with some timeouted variant did not make the issue disappear.
The improved reproducer test can be seen in this branch: https://github.com/ppalaga/camel/commits/CAMEL-19487

Note that it is not always failing. On my machine, I see it failing like 1:10. Invoking it in a loop eventually leads to a failure:

git fetch [e-mail]:ppalaga/camel.git CAMEL-19487 
git checkout -b CAMEL-19487 FETCH_HEAD
# rebuild the tree
mvnd clean install -Pfastinstall

cd core/camel-core
set -e; while true; do mvnd test -Dtest=DynamicRouterConcurrentPOJOTest#testConcurrentDynamicRouter; done

The second commit https://github.com/ppalaga/camel/commits/CAMEL-19487 adds some debug output.
In case of failure it shows that one of the dynamic router instances sometimes sends one or more messages to the wrong endpoint.
Here is an example:

[INFO] Running org.apache.camel.processor.dynamicrouter.DynamicRouterConcurrentPOJOTest
...
[INFO] [stdout] ==== pipe a before Message from seda:a 74
...
[INFO] [stdout] ======= adding to mock:b: Message from seda:a 74
...
[INFO] [stdout] ==== pipe a after Message from seda:a 74 cnt 74
...
[ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.328 s <<< FAILURE! -- in org.apache.camel.processor.dynamicrouter.DynamicRouterConcurrentPOJOTest
[ERROR] org.apache.camel.processor.dynamicrouter.DynamicRouterConcurrentPOJOTest.testConcurrentDynamicRouter -- Time elapsed: 0.317 s <<< FAILURE!
java.lang.AssertionError: 
[Expected mock:b to contain only messages from seda:b, but there were also messages from seda:a] 
Expecting empty but was: ["Message from seda:a 74"]

In this case the dynamic router instance attached to route a sends the message to mock:b.
I'd say that's a serious bug.

This is exactly what the original issue CAMEL-9311 was about.

Observation:
The issue does not occur when each route uses a different dynamic router type: when I copy MyDynamicRouterPojo as MyDynamicRouterPojoB and use it in the route b, then all works as expected.

Issue Links

• is duplicated by: CAMEL-19759 performance: Camel is hanging up during shutdown
• is related to: CAMEL-20035 Program terminates with OutOfMemoryError
• relates to: CAMEL-19425 camel-core: DynamicRouterConcurrentPOJOTest is flaky
• links to: GitHub Pull Request #10948
• links to: GitHub Pull Request #10950
• links to: GitHub Pull Request #10953