[CAMEL-17544] ServicePool.doStop still hangs during shutdown

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.14.0, 3.18.0
• Fix Version/s: 3.21.0, 4.0.0
• Component/s: came-core
• Estimated Complexity: Unknown
• Assignee: Rhuan Rocha
• Reporter: Krzysztof Jamróz
• Created: 24/Jan/22 15:59
• Updated: 17/Jul/23 11:16
• Resolved: 17/Jul/23 11:16

Description

ServicePool.doStop still hangs during shutdown with optimized fix of CAMEL-17536. LinkedHashMap in cache is corrupted not only because of race condtion between acquire and doStop but also between concurrent invocations of acquire during parallel routing of exchanges (see attached repeatTestRecipientList).

 
The root cause is that LinkedHashMap used by SimpleLRUCache is not thread-safe but access is not synchronized. Even reads may modify it as it has LRU policy. And access to LinkedHashMap/{{ SimpleLRUCache }} is possible during routing (so concurrently) not only during start/stop.

 

Uses of SimpleLRUCache in other places in Camel may also exhibit race conditions. It is hard to demonstrate them reliably but I have another example (repeatEndpointsTest). This one usually crashes due to OutOfMemory when converting corrupted (looping) SimpleLRUCache in  EndpointRegistry to array, but I got also other exceptions.

 

java.lang.OutOfMemoryError: Java heap space
    at java.base/java.util.Arrays.copyOf(Arrays.java:3480)
    at java.base/java.util.AbstractCollection.finishToArray(AbstractCollection.java:227)
    at java.base/java.util.AbstractCollection.toArray(AbstractCollection.java:148)
    at java.base/java.util.ArrayList.<init>(ArrayList.java:181)
    at org.apache.camel.impl.engine.AbstractCamelContext.shutdownServices(AbstractCamelContext.java:3582)
    at org.apache.camel.impl.engine.AbstractCamelContext.shutdownServices(AbstractCamelContext.java:3576)
    at org.apache.camel.impl.engine.AbstractCamelContext.doStop(AbstractCamelContext.java:3414)
    at org.apache.camel.support.service.BaseService.stop(BaseService.java:160)
    at org.apache.camel.impl.engine.AbstractCamelContext.stop(AbstractCamelContext.java:2658)
    at org.apache.camel.component.http.ServicePoolShutdownTest.testEndpoints(ServicePoolShutdownTest.java:129)
    at org.apache.camel.component.http.ServicePoolShutdownTest.repeatEndpointsTest(ServicePoolShutdownTest.java:135)

 

Caffeine, which used to be default cache implementation until CAMEL-16093 is thread-safe. Enabling caffeine-lrucache might be a workaround.

Issue Links

• is related to: CAMEL-17536 ServicePool.doStop hangs during shutdown
• relates to: CAMEL-19311 Review the SimpleLRUCache to become thread-safe