Details
Description
As discussed on users@camel.apache.org, i am opening a Jira issue +
We ran into the following issue when dynamically adding and removing ftp routes, whose consumers sometimes take a longer time to start up:
Deadlock when concurrently adding and removing routes
- Default/Abstract)CamelContext.addRoutes -> DefaultCamelContext.addRouteDefinitions locks DefaultModel first, and is then blocked trying to lock DefaultCamelContext
- (Default/Abstract)CamelContext.removeRoute locks DefaultCamelContext first, and is then blocked trying to lock DefaultModel
Threaddump:
Found one Java-level deadlock:
=============================
"pool-2-thread-1":
waiting to lock monitor 0x00007fd22000b0c0 (object 0x000000068c800170, a org.apache.camel.impl.DefaultModel),
which is held by "pool-2-thread-3""pool-2-thread-3":
waiting to lock monitor 0x00007fd21404a130 (object 0x000000068c002a20, a org.apache.camel.impl.DefaultCamelContext),
which is held by "pool-2-thread-2""pool-2-thread-2":
waiting to lock monitor 0x00007fd22000b0c0 (object 0x000000068c800170, a org.apache.camel.impl.DefaultModel),
which is held by "pool-2-thread-3"Java stack information for the threads listed above:
===================================================
"pool-2-thread-1":
at org.apache.camel.impl.DefaultModel.addRouteDefinitions(DefaultModel.java:163)
- waiting to lock <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultCamelContext.addRouteDefinitions(DefaultCamelContext.java:351)at org.apache.camel.builder.RouteBuilder.populateRoutes(RouteBuilder.java:723)at org.apache.camel.builder.RouteBuilder.addRoutesToCamelContext(RouteBuilder.java:569)at org.apache.camel.impl.engine.AbstractCamelContext.addRoutes(AbstractCamelContext.java:1186)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest.lambda$1(DynamicallyConcurrentlyAddRoutesTest.java:64)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest$$Lambda$487/0x00000008002c52d0.call(Unknown Source)at java.util.concurrent.FutureTask.run(java.base@17.0.3/FutureTask.java:264)at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.3/ThreadPoolExecutor.java:1136)at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.3/ThreadPoolExecutor.java:635)at java.lang.Thread.run(java.base@17.0.3/Thread.java:833)"pool-2-thread-3":at org.apache.camel.impl.engine.AbstractCamelContext.removeRoute(AbstractCamelContext.java:1400)
- waiting to lock <0x000000068c002a20> (a org.apache.camel.impl.DefaultCamelContext)at org.apache.camel.impl.DefaultModel.removeRouteDefinition(DefaultModel.java:212)
- locked <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultModel.removeRouteDefinitions(DefaultModel.java:201)
- locked <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultModel.addRouteDefinitions(DefaultModel.java:179)
- locked <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultCamelContext.addRouteDefinitions(DefaultCamelContext.java:351)at org.apache.camel.builder.RouteBuilder.populateRoutes(RouteBuilder.java:723)at org.apache.camel.builder.RouteBuilder.addRoutesToCamelContext(RouteBuilder.java:569)at org.apache.camel.impl.engine.AbstractCamelContext.addRoutes(AbstractCamelContext.java:1186)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest.lambda$1(DynamicallyConcurrentlyAddRoutesTest.java:64)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest$$Lambda$487/0x00000008002c52d0.call(Unknown Source)at java.util.concurrent.FutureTask.run(java.base@17.0.3/FutureTask.java:264)at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.3/ThreadPoolExecutor.java:1136)at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.3/ThreadPoolExecutor.java:635)at java.lang.Thread.run(java.base@17.0.3/Thread.java:833)"pool-2-thread-2":at org.apache.camel.impl.DefaultModel.getRouteDefinition(DefaultModel.java:237)
- waiting to lock <0x000000068c800170> (a org.apache.camel.impl.DefaultModel)at org.apache.camel.impl.DefaultCamelContext.getRouteDefinition(DefaultCamelContext.java:343)at org.apache.camel.impl.DefaultCamelContext.removeRoute(DefaultCamelContext.java:959)
- locked <0x000000068c002a20> (a org.apache.camel.impl.DefaultCamelContext)at org.apache.camel.impl.engine.AbstractCamelContext.removeRoute(AbstractCamelContext.java:1400)
- locked <0x000000068c002a20> (a org.apache.camel.impl.DefaultCamelContext)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest.lambda$1(DynamicallyConcurrentlyAddRoutesTest.java:77)at org.apache.camel.impl.DynamicallyConcurrentlyAddRoutesTest$$Lambda$487/0x00000008002c52d0.call(Unknown Source)at java.util.concurrent.FutureTask.run(java.base@17.0.3/FutureTask.java:264)at java.util.concurrent.ThreadPoolExecutor.runWorker(java.base@17.0.3/ThreadPoolExecutor.java:1136)at java.util.concurrent.ThreadPoolExecutor$Worker.run(java.base@17.0.3/ThreadPoolExecutor.java:635)at java.lang.Thread.run(java.base@17.0.3/Thread.java:833)
Found 1 deadlock.
My JUnit test executed successfully when I changed DefaultCamelContext to obtain the lock on model before locking camelContext in removeRoute.
Attachments
Issue Links
- links to