[BROOKLYN-404] ConcurrentModificationException on rebind to persisted state (in BrooklynProperties)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 0.11.0
• Assignee: Aled Sage
• Reporter: Aled Sage
• Created: 25/Nov/16 15:16
• Updated: 24/Jan/17 15:44
• Resolved: 24/Jan/17 15:44

Description

With Brooklyn 0.10.0-SNAPSHOT, when attempting to restart Brooklyn (binding to existing persisted state), it hit a ConcurrentModificationException.

This failure only happens occasionally.

Starting Brooklyn such that it fails fast if there are any persistence errors:

./bin/brooklyn launch --persist auto  --persistenceDir `pwd`/$PERSISTENCE_DIR/data --highAvailability disabled --startupFailOnPersistenceErrors --startupFailOnManagedAppsErrors

The info log showed:

2016-11-24 09:49:31,612 ERROR Startup error: Error rebinding to persisted state: Failure rebinding: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)]): ConcurrentModificationException
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: 
	at org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:129) ~[brooklyn-utils-common-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
Caused by: java.util.concurrent.ExecutionException: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Failure rebinding: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)]): ConcurrentModificationException
	at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_111]
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Failure rebinding: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)]): ConcurrentModificationException
	at org.apache.brooklyn.util.exceptions.Exceptions.create(Exceptions.java:432) ~[brooklyn-utils-common-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
Caused by: java.lang.IllegalStateException: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)])
	at org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onRebindFailed(RebindExceptionHandlerImpl.java:304) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
Caused by: java.util.ConcurrentModificationException: null
	at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719) ~[na:1.8.0_111]
Startup error: Error rebinding to persisted state: Failure rebinding: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)]): ConcurrentModificationException

The debug log showed:

2016-11-24 09:49:31,610 DEBUG o.a.b.l.common.BasicLauncher [main]: Error rebinding to persisted state (rethrowing): org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Failure rebinding: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)]): ConcurrentModificationException
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: 
        at org.apache.brooklyn.util.exceptions.Exceptions.propagate(Exceptions.java:129) ~[brooklyn-utils-common-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebind(RebindManagerImpl.java:513) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.launcher.common.BasicLauncher.startPersistenceWithoutHA(BasicLauncher.java:662) [brooklyn-launcher-common-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.launcher.common.BasicLauncher.startPersistence(BasicLauncher.java:624) [brooklyn-launcher-common-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.launcher.common.BasicLauncher.handlePersistence(BasicLauncher.java:511) [brooklyn-launcher-common-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.launcher.common.BasicLauncher.start(BasicLauncher.java:429) [brooklyn-launcher-common-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.cli.Main$LaunchCommand.call(Main.java:462) [brooklyn-cli-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.cli.Main$LaunchCommand.call(Main.java:197) [brooklyn-cli-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.cli.AbstractMain.execCli(AbstractMain.java:252) [brooklyn-cli-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.cli.AbstractMain.execCli(AbstractMain.java:244) [brooklyn-cli-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at io.cloudsoft.amp.Main.main(Main.java:39) [amp-launcher-3.5.1-20161121.1656.jar:na]
Caused by: java.util.concurrent.ExecutionException: org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Failure rebinding: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)]): ConcurrentModificationException
        at java.util.concurrent.FutureTask.report(FutureTask.java:122) ~[na:1.8.0_111]
        at java.util.concurrent.FutureTask.get(FutureTask.java:192) ~[na:1.8.0_111]
        at com.google.common.util.concurrent.ForwardingFuture.get(ForwardingFuture.java:63) ~[guava-16.0.1.jar:na]
        at org.apache.brooklyn.util.core.task.BasicTask.get(BasicTask.java:361) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebind(RebindManagerImpl.java:511) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        ... 9 common frames omitted
org.apache.brooklyn.util.exceptions.PropagatedRuntimeException: Failure rebinding: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)]): ConcurrentModificationException
        at org.apache.brooklyn.util.exceptions.Exceptions.create(Exceptions.java:432) ~[brooklyn-utils-common-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onDoneImpl(RebindExceptionHandlerImpl.java:497) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onDone(RebindExceptionHandlerImpl.java:413) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.run(RebindIteration.java:268) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl.rebindImpl(RebindManagerImpl.java:558) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:508) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindManagerImpl$3.call(RebindManagerImpl.java:506) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.util.core.task.BasicExecutionManager$SubmissionCallable.call(BasicExecutionManager.java:522) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_111]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[na:1.8.0_111]
        at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_111]
Caused by: java.lang.IllegalStateException: problem rebinding location cMBNYzIx (SshMachineLocation[SshMachineLocation:cMBN:ampqa@54.184.92.166/54.184.92.166:22(id=cMBNYzIx)])
        at org.apache.brooklyn.core.mgmt.rebind.RebindExceptionHandlerImpl.onRebindFailed(RebindExceptionHandlerImpl.java:304) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.reconstructEverything(RebindIteration.java:543) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.doRun(RebindIteration.java:243) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.InitialFullRebindIteration.doRun(InitialFullRebindIteration.java:69) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.run(RebindIteration.java:266) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        ... 8 common frames omitted
Caused by: java.util.ConcurrentModificationException: null
        at java.util.LinkedHashMap$LinkedHashIterator.nextNode(LinkedHashMap.java:719) ~[na:1.8.0_111]
        at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:752) ~[na:1.8.0_111]
        at java.util.LinkedHashMap$LinkedEntryIterator.next(LinkedHashMap.java:750) ~[na:1.8.0_111]
        at org.apache.brooklyn.core.internal.BrooklynPropertiesImpl.getAllConfigLocalRaw(BrooklynPropertiesImpl.java:463) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.internal.BrooklynPropertiesImpl.getAllConfig(BrooklynPropertiesImpl.java:470) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.internal.DeferredBrooklynProperties.asMapWithStringKeys(DeferredBrooklynProperties.java:162) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.location.BasicLocationRegistry.getProperties(BasicLocationRegistry.java:516) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.location.AbstractLocationResolver.extractConfig(AbstractLocationResolver.java:101) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.location.access.PortForwardManagerLocationResolver.extractConfig(PortForwardManagerLocationResolver.java:84) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.location.access.PortForwardManagerLocationResolver.newLocationSpecFromString(PortForwardManagerLocationResolver.java:60) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.location.BasicLocationRegistry.getLocationSpec(BasicLocationRegistry.java:375) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.location.BasicLocationRegistry.getLocationSpec(BasicLocationRegistry.java:355) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.location.BasicLocationRegistry.getLocationManaged(BasicLocationRegistry.java:331) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.location.ssh.SshMachineLocation.init(SshMachineLocation.java:306) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.BasicLocationRebindSupport.addCustoms(BasicLocationRebindSupport.java:108) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.BasicLocationRebindSupport.addCustoms(BasicLocationRebindSupport.java:38) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.AbstractBrooklynObjectRebindSupport.reconstruct(AbstractBrooklynObjectRebindSupport.java:65) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        at org.apache.brooklyn.core.mgmt.rebind.RebindIteration.reconstructEverything(RebindIteration.java:541) ~[brooklyn-core-0.10.0-20161121.1656.jar:0.10.0-20161121.1656]
        ... 11 common frames omitted

Issue Links

• links to: GitHub Pull Request #505