[BROOKLYN-245] Deadlock from ControlledDynamicWebAppCluster, plus thread modifying attribute

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.9.0
• Assignee: Aled Sage
• Reporter: Aled Sage
• Created: 31/Mar/16 19:44
• Updated: 06/Apr/16 16:18
• Resolved: 06/Apr/16 16:18

Description

With master (i.e. would also affect 0.9.0)...

When running org.apache.brooklyn.qa.load.LoadTest several times, I hit the deadlock below.

The test deploys several apps concurrently, each using ControlledDynamicWebAppCluster (via the Java-app SimulatedTheeTierApp). This deadlock could happen in other uses of ControlledDynamicWebAppCluster, or in other entities depending if/when they synchronize on the entity themselves.

Java stack information for the threads listed above:
===================================================
"brooklyn-execmanager-H7Adif6C-157":
        at org.apache.brooklyn.core.sensor.AttributeMap.asMap(AttributeMap.java:92)
        - waiting to lock <0x00000007d3504b60> (a java.util.Collections$SynchronizedMap)
        at org.apache.brooklyn.core.entity.AbstractEntity$BasicSensorSupport.getAll(AbstractEntity.java:1129)
        at org.apache.brooklyn.core.entity.AbstractEntity.getAllAttributes(AbstractEntity.java:1003)
        at org.apache.brooklyn.core.mgmt.rebind.dto.MementosGenerators.newEntityMementoBuilder(MementosGenerators.java:194)
        at org.apache.brooklyn.core.mgmt.rebind.BasicEntityRebindSupport.getMementoWithProperties(BasicEntityRebindSupport.java:69)
        at org.apache.brooklyn.core.mgmt.rebind.BasicEntityRebindSupport.getMemento(BasicEntityRebindSupport.java:61)
        at org.apache.brooklyn.core.mgmt.rebind.BasicEntityRebindSupport.getMemento(BasicEntityRebindSupport.java:1)
        at org.apache.brooklyn.core.mgmt.rebind.PeriodicDeltaChangeListener.persistNowInternal(PeriodicDeltaChangeListener.java:417)
        at org.apache.brooklyn.core.mgmt.rebind.PeriodicDeltaChangeListener.persistNowSafely(PeriodicDeltaChangeListener.java:352)
        at org.apache.brooklyn.core.mgmt.rebind.PeriodicDeltaChangeListener.persistNowSafely(PeriodicDeltaChangeListener.java:346)
        at org.apache.brooklyn.core.mgmt.rebind.PeriodicDeltaChangeListener$1$1.call(PeriodicDeltaChangeListener.java:215)
        at org.apache.brooklyn.core.mgmt.rebind.PeriodicDeltaChangeListener$1$1.call(PeriodicDeltaChangeListener.java:1)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$ScheduledTaskCallable$1.call(BasicExecutionManager.java:436)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
"brooklyn-execmanager-H7Adif6C-16":
        at org.apache.brooklyn.core.entity.AbstractEntity$BasicSubscriptionSupport.getSubscriptionContext(AbstractEntity.java:1488)
        - waiting to lock <0x00000007d34abc20> (a org.apache.brooklyn.entity.webapp.ControlledDynamicWebAppClusterImpl)
        at org.apache.brooklyn.core.entity.AbstractEntity$BasicSensorSupport.emitInternal(AbstractEntity.java:1164)
        at org.apache.brooklyn.core.entity.AbstractEntity.emitInternal(AbstractEntity.java:2021)
        at org.apache.brooklyn.core.sensor.AttributeMap.update(AttributeMap.java:133)
        at org.apache.brooklyn.core.sensor.AttributeMap.modify(AttributeMap.java:162)
        - locked <0x00000007d3504b60> (a java.util.Collections$SynchronizedMap)
        at org.apache.brooklyn.core.entity.AbstractEntity$BasicSensorSupport.modify(AbstractEntity.java:1106)
        at org.apache.brooklyn.core.entity.lifecycle.ServiceStateLogic.updateMapSensorEntry(ServiceStateLogic.java:150)
        at org.apache.brooklyn.core.entity.lifecycle.ServiceStateLogic.clearMapSensorEntry(ServiceStateLogic.java:102)
        at org.apache.brooklyn.core.entity.lifecycle.ServiceStateLogic$ComputeServiceIndicatorsFromChildrenAndMembers.updateMapSensor(ServiceStateLogic.java:558)
        at org.apache.brooklyn.core.entity.lifecycle.ServiceStateLogic$ComputeServiceIndicatorsFromChildrenAndMembers.onUpdated(ServiceStateLogic.java:477)
        at org.apache.brooklyn.enricher.stock.AbstractAggregator$8.onEvent(AbstractAggregator.java:178)
        at org.apache.brooklyn.core.mgmt.internal.LocalSubscriptionManager$2.run(LocalSubscriptionManager.java:244)
        at org.apache.brooklyn.util.concurrent.CallableFromRunnable.call(CallableFromRunnable.java:43)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
        at org.apache.brooklyn.util.core.task.SingleThreadedScheduler$1.call(SingleThreadedScheduler.java:116)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)
"brooklyn-execmanager-H7Adif6C-4":
        at java.util.Collections$SynchronizedMap.get(Collections.java:2037)
        - waiting to lock <0x00000007d3504b60> (a java.util.Collections$SynchronizedMap)
        at org.apache.brooklyn.core.sensor.AttributeMap.getValue(AttributeMap.java:200)
        at org.apache.brooklyn.core.sensor.AttributeMap.getValue(AttributeMap.java:206)
        at org.apache.brooklyn.core.entity.AbstractEntity$BasicSensorSupport.get(AbstractEntity.java:1038)
        at org.apache.brooklyn.core.entity.AbstractEntity.getAttribute(AbstractEntity.java:946)
        at org.apache.brooklyn.entity.webapp.ControlledDynamicWebAppClusterImpl.getCluster(ControlledDynamicWebAppClusterImpl.java:194)
        - locked <0x00000007d34abc20> (a org.apache.brooklyn.entity.webapp.ControlledDynamicWebAppClusterImpl)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.brooklyn.core.objs.proxy.EntityProxyImpl.invoke(EntityProxyImpl.java:213)
        at com.sun.proxy.$Proxy57.getCluster(Unknown Source)
        at org.apache.brooklyn.qa.load.SimulatedTheeTierApp.init(SimulatedTheeTierApp.java:102)
        at org.apache.brooklyn.core.objs.proxy.InternalEntityFactory$1.run(InternalEntityFactory.java:327)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:519)
        at java.util.concurrent.FutureTask.run(FutureTask.java:262)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:745)

Found 1 deadlock.