[CAMEL-18472] deadlock when concurrently adding and removing routes

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.7.7, 3.19.0
• Fix Version/s: 3.19.0
• Component/s: came-core
• Estimated Complexity: Unknown
• Assignee: Unassigned
• Reporter: Gerda Ilger
• Created: 06/Sep/22 15:30
• Updated: 08/Sep/22 11:38
• Resolved: 08/Sep/22 11:38

Description

As discussed on [e-mail], i am opening a Jira issue +

We ran into the following issue when dynamically adding and removing ftp routes, whose consumers sometimes take a longer time to start up:

Deadlock when concurrently adding and removing routes

• Default/Abstract)CamelContext.addRoutes -> DefaultCamelContext.addRouteDefinitions locks DefaultModel first, and is then blocked trying to lock DefaultCamelContext

• (Default/Abstract)CamelContext.removeRoute locks DefaultCamelContext first, and is then blocked trying to lock DefaultModel

Threaddump:

Found one Java-level deadlock:
=============================
"pool-2-thread-1":
waiting to lock monitor 0x00007fd22000b0c0 (object 0x000000068c800170, a org.apache.camel.impl.DefaultModel),
which is held by "pool-2-thread-3""pool-2-thread-3":
waiting to lock monitor 0x00007fd21404a130 (object 0x000000068c002a20, a org.apache.camel.impl.DefaultCamelContext),
which is held by "pool-2-thread-2""pool-2-thread-2":
waiting to lock monitor 0x00007fd22000b0c0 (object 0x000000068c800170, a org.apache.camel.impl.DefaultModel),
which is held by "pool-2-thread-3"Java stack information for the threads listed above:
===================================================
"pool-2-thread-1":
at org.apache.camel.impl.DefaultModel.addRouteDefinitions(DefaultModel.java:163)

• waiting to lock <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultCamelContext.addRouteDefinitions(DefaultCamelContext.java:351)at org.apache.camel.builder.RouteBuilder.populateRoutes(RouteBuilder.java:723)at org.apache.camel.builder.RouteBuilder.addRoutesToCamelContext(RouteBuilder.java:569)at org.apache.camel.impl.engine.AbstractCamelContext.addRoutes(AbstractCamelContext.java:1186)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest.lambda$1(DynamicallyConcurrentlyAddRoutesTest.java:64)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest$$Lambda$487/0x00000008002c52d0.call(Unknown Source)at java.util.concurrent.FutureTask.run(java.base@17.0.3/FutureTask.java:264)at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.3/ThreadPoolExecutor.java:1136)at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.3/ThreadPoolExecutor.java:635)at java.lang.Thread.run(java.base@17.0.3/Thread.java:833)"pool-2-thread-3":at org.apache.camel.impl.engine.AbstractCamelContext.removeRoute(AbstractCamelContext.java:1400)
• waiting to lock <0x000000068c002a20> (a org.apache.camel.impl.DefaultCamelContext)at org.apache.camel.impl.DefaultModel.removeRouteDefinition(DefaultModel.java:212)
• locked <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultModel.removeRouteDefinitions(DefaultModel.java:201)
• locked <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultModel.addRouteDefinitions(DefaultModel.java:179)
• locked <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultCamelContext.addRouteDefinitions(DefaultCamelContext.java:351)at org.apache.camel.builder.RouteBuilder.populateRoutes(RouteBuilder.java:723)at org.apache.camel.builder.RouteBuilder.addRoutesToCamelContext(RouteBuilder.java:569)at org.apache.camel.impl.engine.AbstractCamelContext.addRoutes(AbstractCamelContext.java:1186)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest.lambda$1(DynamicallyConcurrentlyAddRoutesTest.java:64)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest$$Lambda$487/0x00000008002c52d0.call(Unknown Source)at java.util.concurrent.FutureTask.run(java.base@17.0.3/FutureTask.java:264)at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.3/ThreadPoolExecutor.java:1136)at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.3/ThreadPoolExecutor.java:635)at java.lang.Thread.run(java.base@17.0.3/Thread.java:833)"pool-2-thread-2":at org.apache.camel.impl.DefaultModel.getRouteDefinition(DefaultModel.java:237)
• waiting to lock <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultCamelContext.getRouteDefinition(DefaultCamelContext.java:343)at org.apache.camel.impl.DefaultCamelContext.removeRoute(DefaultCamelContext.java:959)
• locked <0x000000068c002a20> (a org.apache.camel.impl.DefaultCamelContext)at org.apache.camel.impl.engine.AbstractCamelContext.removeRoute(AbstractCamelContext.java:1400)
• locked <0x000000068c002a20> (a org.apache.camel.impl.DefaultCamelContext)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest.lambda$1(DynamicallyConcurrentlyAddRoutesTest.java:77)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest$$Lambda$487/0x00000008002c52d0.call(Unknown Source)at java.util.concurrent.FutureTask.run(java.base@17.0.3/FutureTask.java:264)at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.3/ThreadPoolExecutor.java:1136)at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.3/ThreadPoolExecutor.java:635)at java.lang.Thread.run(java.base@17.0.3/Thread.java:833)

Found 1 deadlock.

My JUnit test executed successfully when I changed DefaultCamelContext to obtain the lock on model before locking camelContext in removeRoute.

Issue Links

• links to: GitHub Pull Request #8303