[CAMEL-6941] ConcurrentModificationException while shutting down camel

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.11.0
• Fix Version/s: 2.11.3, 2.12.2, 2.13.0
• Component/s: camel-core

Environment

jdk-1.7.0_45

• Estimated Complexity: Unknown
• Assignee: Claus Ibsen
• Reporter: Robin Kåveland
• Created: 06/Nov/13 09:43
• Updated: 08/Nov/13 14:51
• Resolved: 08/Nov/13 14:51

Description

Observed the following stacktrace while shutting down camel:

   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 4.017 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 6.027 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 8.037 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 10.008 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
g   org.apache.camel.impl.DefaultExecutorServiceManager.doShutdown()#301: Forcing shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda due first await termination elapsed.
   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 2.008 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 4.018 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 6.028 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 8.039 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
   org.apache.camel.impl.DefaultExecutorServiceManager.awaitTermination()#401: Waited 10.008 seconds for ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda to terminate...
g   org.apache.camel.impl.DefaultExecutorServiceManager.doShutdown()#305: Cannot completely force shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda due second await termination elapsed.
   org.apache.camel.impl.DefaultExecutorServiceManager.doShutdown()#318: Shutdown of ExecutorService: java.util.concurrent.ThreadPoolExecutor@33c4bcda is shutdown: true and terminated: false took: 20.018 seconds.
g   org.apache.camel.impl.DefaultCamelContext.shutdownServices()#1771: Error occurred while shutting down service: org.apache.camel.impl.DefaultExecutorServiceManager@37285252. This exception will be ignored.
g   Caused by java.util.ConcurrentModificationException - null
g      at java.util.AbstractList$Itr.checkForComodification(AbstractList.java:372)
g      at java.util.AbstractList$Itr.next(AbstractList.java:343)
g      at org.apache.camel.impl.DefaultExecutorServiceManager.doShutdown(DefaultExecutorServiceManager.java:442)
g      at org.apache.camel.support.ServiceSupport.shutdown(ServiceSupport.java:158)
g      at org.apache.camel.util.ServiceHelper.stopAndShutdownService(ServiceHelper.java:166)
g      at org.apache.camel.impl.DefaultCamelContext.shutdownServices(DefaultCamelContext.java:1766)
g      at org.apache.camel.impl.DefaultCamelContext.doStop(DefaultCamelContext.java:1681)
g      at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:92)
g      at org.apache.camel.core.xml.AbstractCamelContextFactoryBean.destroy(AbstractCamelContextFactoryBean.java:432)
g      at org.springframework.beans.factory.support.DisposableBeanAdapter.destroy(DisposableBeanAdapter.java:211)
g      at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroyBean(DefaultSingletonBeanRegistry.java:500)
g      at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingleton(DefaultSingletonBeanRegistry.java:476)
g      at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.destroySingletons(DefaultSingletonBeanRegistry.java:445)
g      at org.springframework.context.support.AbstractApplicationContext.destroyBeans(AbstractApplicationContext.java:1090)
g      at org.springframework.context.support.AbstractApplicationContext.doClose(AbstractApplicationContext.java:1064)
g      at org.springframework.context.support.AbstractApplicationContext.close(AbstractApplicationContext.java:1010)
g      at org.apache.camel.spring.Main.doStop(Main.java:152)
g      at org.apache.camel.support.ServiceSupport.stop(ServiceSupport.java:92)
g      at org.apache.camel.main.MainSupport$HangupInterceptor.run(MainSupport.java:82)
   org.apache.camel.impl.DefaultCamelContext.doStop()#1695: Uptime 35 days 23 hours
   org.apache.camel.impl.DefaultCamelContext.doStop()#1696: Apache Camel 2.11.0 (CamelContext: camel-activemq) is shutdown in 51.196 seconds

I haven't seen this before but it seems to me that there must be more than one thread doing something to DefaultExecutorServiceManager.executorServices (ArrayList) at the same time.