[ARIES-1051] Blueprint deadlock occurs during Karaf startup or bundle update action

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: blueprint-core-1.1.0
• Component/s: Blueprint

Environment

Windows 7, x64

java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

Karaf 2.3.1

• Assignee: Unassigned
• Reporter: Dmitry Konstantinov
• Created: 18/Apr/13 07:20
• Updated: 13/Jul/15 08:01

Description

There are two locks - felix bundle lock and lock of AtomicBoolean scheduled in BlueprintContainerImpl class and these locks can be occured in diferent order for some situations.
For example:
1) In case of container initialization the order is following: blueprint lock (inside executor task) -> bundle lock (inside service registration)
2) In case of container destroying: bundle lock (bundle tracker) -> blueprint lock (BlueprintContainerImpl.namespaceHandlerUnregistered method)

The thread dump provides a real example of the deadlock:

"Blueprint Extender: 3" daemon prio=6 tid=0x0000000006ec4000 nid=0x1658 in Object.wait() [0x0000000009b2e000]
   java.lang.Thread.State: WAITING (on object monitor)
	at java.lang.Object.wait(Native Method)
	- waiting on <0x00000000e098fdf0> (a [Ljava.lang.Object;)
	at java.lang.Object.wait(Object.java:485)
	at org.apache.felix.framework.Felix.acquireBundleLock(Felix.java:4871)
	- locked <0x00000000e098fdf0> (a [Ljava.lang.Object;)
	at org.apache.felix.framework.Felix.registerService(Felix.java:3205)
	at org.apache.felix.framework.BundleContextImpl.registerService(BundleContextImpl.java:346)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.registerService(BlueprintContainerImpl.java:439)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.doRun(BlueprintContainerImpl.java:379)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.run(BlueprintContainerImpl.java:261)
	- locked <0x00000000e0274b50> (a java.util.concurrent.atomic.AtomicBoolean)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at org.apache.aries.blueprint.container.ExecutorServiceWrapper.run(ExecutorServiceWrapper.java:106)
	at org.apache.aries.blueprint.utils.threading.impl.DiscardableRunnable.run(DiscardableRunnable.java:48)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:98)
	at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:206)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)




"FelixFrameworkWiring" daemon prio=6 tid=0x0000000006eca000 nid=0x900 waiting for monitor entry [0x000000000b30e000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.namespaceHandlerUnregistered(BlueprintContainerImpl.java:892)
	- waiting to lock <0x00000000e0274b50> (a java.util.concurrent.atomic.AtomicBoolean)
	at org.apache.aries.blueprint.namespace.NamespaceHandlerRegistryImpl$NamespaceHandlerSetImpl.unregisterHandler(NamespaceHandlerRegistryImpl.java:582)
	at org.apache.aries.blueprint.namespace.NamespaceHandlerRegistryImpl.unregisterHandler(NamespaceHandlerRegistryImpl.java:173)
	at org.apache.aries.blueprint.namespace.NamespaceHandlerRegistryImpl.removedService(NamespaceHandlerRegistryImpl.java:144)
	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:956)
	at org.osgi.util.tracker.ServiceTracker$Tracked.customizerRemoved(ServiceTracker.java:1)
	at org.osgi.util.tracker.AbstractTracked.untrack(AbstractTracked.java:341)
	at org.osgi.util.tracker.ServiceTracker$Tracked.serviceChanged(ServiceTracker.java:902)
	at org.apache.felix.framework.util.EventDispatcher.invokeServiceListenerCallback(EventDispatcher.java:932)
	at org.apache.felix.framework.util.EventDispatcher.fireEventImmediately(EventDispatcher.java:793)
	at org.apache.felix.framework.util.EventDispatcher.fireServiceEvent(EventDispatcher.java:543)
	at org.apache.felix.framework.Felix.fireServiceEvent(Felix.java:4260)
	at org.apache.felix.framework.Felix.access$000(Felix.java:74)
	at org.apache.felix.framework.Felix$1.serviceChanged(Felix.java:390)
	at org.apache.felix.framework.ServiceRegistry.unregisterService(ServiceRegistry.java:148)
	at org.apache.felix.framework.ServiceRegistrationImpl.unregister(ServiceRegistrationImpl.java:127)
	at org.apache.aries.util.AriesFrameworkUtil.safeUnregisterService(AriesFrameworkUtil.java:114)
	at org.apache.aries.blueprint.container.ServiceRecipe.unregister(ServiceRecipe.java:209)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.unregisterServices(BlueprintContainerImpl.java:701)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.tidyupComponents(BlueprintContainerImpl.java:907)
	at org.apache.aries.blueprint.container.BlueprintContainerImpl.destroy(BlueprintContainerImpl.java:857)
	at org.apache.aries.blueprint.container.BlueprintExtender$3.run(BlueprintExtender.java:284)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
	at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
	at java.util.concurrent.FutureTask.run(FutureTask.java:138)
	at org.apache.aries.blueprint.container.BlueprintExtender.destroyContainer(BlueprintExtender.java:305)
	at org.apache.aries.blueprint.container.BlueprintExtender.modifiedBundle(BlueprintExtender.java:206)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:500)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.customizerModified(BundleHookBundleTracker.java:433)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$AbstractTracked.track(BundleHookBundleTracker.java:725)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$Tracked.bundleChanged(BundleHookBundleTracker.java:463)
	at org.apache.aries.util.tracker.hook.BundleHookBundleTracker$BundleEventHook.event(BundleHookBundleTracker.java:422)
	at org.apache.felix.framework.util.SecureAction.invokeBundleEventHook(SecureAction.java:1103)
	at org.apache.felix.framework.util.EventDispatcher.createWhitelistFromHooks(EventDispatcher.java:695)
	at org.apache.felix.framework.util.EventDispatcher.fireBundleEvent(EventDispatcher.java:483)
	at org.apache.felix.framework.Felix.fireBundleEvent(Felix.java:4244)
	at org.apache.felix.framework.Felix.stopBundle(Felix.java:2351)
	at org.apache.felix.framework.Felix$RefreshHelper.stop(Felix.java:4629)
	at org.apache.felix.framework.Felix.refreshPackages(Felix.java:3951)
	at org.apache.felix.framework.FrameworkWiringImpl.run(FrameworkWiringImpl.java:172)
	at java.lang.Thread.run(Thread.java:662)

Issue Links

• is broken by: ARIES-986 The blueprint extender does not support high concurrency very well
• relates to: ARIES-896 Deadlock with blueprint services
• relates to: KARAF-1724 Blueprint bundle deadlocks in "Stopping" status when update-ing/dev:watch bundle
• relates to: FELIX-3067 Prevent Deadlock Situation in Felix.acquireGlobalLock