[IGNITE-3024] Possible deadlock in services when Spring is used

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.5.0.final
• Fix Version/s: 1.7
• Component/s: managed services
• Labels: community customer important
• Flags: Important
• Assignee: Unassigned
• Reporter: Valentin Kulichenko
• Created: 18/Apr/16 23:32
• Updated: 22/Jul/16 13:10
• Resolved: 22/Jul/16 13:10

Description

Resource injection in service processor happens inside synchronized block (see redeploy() method). During injection Spring will acquire its global bean factory lock:

"srvc-deploy-#54%xxx%":
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:187)
	- waiting to lock <0x00000006c03e2290> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:173)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:240)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
	at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:956)
	at org.apache.ignite.internal.processors.resource.GridResourceSpringBeanInjector.inject(GridResourceSpringBeanInjector.java:65)
	at org.apache.ignite.internal.processors.resource.GridResourceIoc.injectInternal(GridResourceIoc.java:172)
	at org.apache.ignite.internal.processors.resource.GridResourceIoc.inject(GridResourceIoc.java:97)
	at org.apache.ignite.internal.processors.resource.GridResourceProcessor.inject(GridResourceProcessor.java:510)
	at org.apache.ignite.internal.processors.service.GridServiceProcessor.copyAndInject(GridServiceProcessor.java:957)
	at org.apache.ignite.internal.processors.service.GridServiceProcessor.redeploy(GridServiceProcessor.java:873)
	- locked <0x000000077ffd4e50> (a java.util.ArrayList)
	at org.apache.ignite.internal.processors.service.GridServiceProcessor.access$4400(GridServiceProcessor.java:94)
	at org.apache.ignite.internal.processors.service.GridServiceProcessor$AssignmentListener$1.run0(GridServiceProcessor.java:1348)
	at org.apache.ignite.internal.processors.service.GridServiceProcessor$BusyRunnable.run(GridServiceProcessor.java:1399)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

If some Spring bean creates a service proxy during initialization, we can get a deadlock:

"localhost-startStop-1":
	at org.apache.ignite.internal.processors.service.GridServiceProcessor.serviceContext(GridServiceProcessor.java:613)
	- waiting to lock <0x000000077ffd4e50> (a java.util.ArrayList)
	at org.apache.ignite.internal.processors.service.GridServiceProcessor.serviceProxy(GridServiceProcessor.java:633)
	at org.apache.ignite.internal.IgniteServicesImpl.serviceProxy(IgniteServicesImpl.java:232)
	...
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:497)
	at org.springframework.beans.factory.support.SimpleInstantiationStrategy.instantiate(SimpleInstantiationStrategy.java:162)
	at org.springframework.beans.factory.support.ConstructorResolver.instantiateUsingFactoryMethod(ConstructorResolver.java:588)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.instantiateUsingFactoryMethod(AbstractAutowireCapableBeanFactory.java:1119)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBeanInstance(AbstractAutowireCapableBeanFactory.java:1014)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:504)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	- locked <0x00000006c03e2290> (a java.util.concurrent.ConcurrentHashMap)
	at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:299)
	at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:194)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.findAutowireCandidates(DefaultListableBeanFactory.java:1120)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.doResolveDependency(DefaultListableBeanFactory.java:1044)
	at org.springframework.beans.factory.support.DefaultListableBeanFactory.resolveDependency(DefaultListableBeanFactory.java:942)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.autowireResource(CommonAnnotationBeanPostProcessor.java:457)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.getResource(CommonAnnotationBeanPostProcessor.java:435)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor$ResourceElement.getResourceToInject(CommonAnnotationBeanPostProcessor.java:559)
	at org.springframework.beans.factory.annotation.InjectionMetadata$InjectedElement.inject(InjectionMetadata.java:169)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
	at org.springframework.context.annotation.CommonAnnotationBeanPostProcessor.postProcessPropertyValues(CommonAnnotationBeanPostProcessor.java:305)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1210)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:537)
	at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:476)
	at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:303)
	at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
	- locked <0x00000006c03e2290> (a java.util.concurrent.ConcurrentHashMap)
	...

We should make sure that injections are not happening inside other locks.