[BROOKLYN-260] ConcurrentModificationException in BrooklynProperties during startup (Brooklyn failed to start)

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.9.0
• Fix Version/s: 0.11.0
• Assignee: Unassigned
• Reporter: Aled Sage
• Created: 03/May/16 17:10
• Updated: 27/Apr/17 22:03
• Resolved: 27/Apr/17 22:03

Description

Running brooklyn launch, it failed to start with the ConcurrentModificationException shown below. Trying a second time, it worked fine. I've only seen this error once.

2016-04-29 15:26:23,873 ERROR o.a.brooklyn.cli.AbstractMain [main]: Execution error: null
java.util.ConcurrentModificationException: null
        at java.util.LinkedHashMap$LinkedHashIterator.nextEntry(LinkedHashMap.java:394) ~[na:1.7.0_71]
        at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:413) ~[na:1.7.0_71]
        at java.util.LinkedHashMap$EntryIterator.next(LinkedHashMap.java:412) ~[na:1.7.0_71]
        at org.apache.brooklyn.core.internal.BrooklynPropertiesImpl.submap(BrooklynPropertiesImpl.java:463) ~[org.apache.brooklyn-brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.core.internal.BrooklynPropertiesImpl.submap(BrooklynPropertiesImpl.java:61) ~[org.apache.brooklyn-brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.core.mgmt.internal.DeferredBrooklynProperties.submap(DeferredBrooklynProperties.java:193) ~[org.apache.brooklyn-brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.core.mgmt.internal.DeferredBrooklynProperties.submap(DeferredBrooklynProperties.java:55) ~[org.apache.brooklyn-brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.core.location.BasicLocationRegistry.updateDefinedLocations(BasicLocationRegistry.java:277) ~[org.apache.brooklyn-brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.core.location.BasicLocationRegistry.<init>(BasicLocationRegistry.java:165) ~[org.apache.brooklyn-brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.core.mgmt.internal.AbstractManagementContext.getLocationRegistry(AbstractManagementContext.java:381) ~[org.apache.brooklyn-brooklyn-core-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.launcher.common.BasicLauncher.addLocations(BasicLauncher.java:473) ~[org.apache.brooklyn-brooklyn-launcher-common-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.launcher.common.BasicLauncher.start(BasicLauncher.java:427) ~[org.apache.brooklyn-brooklyn-launcher-common-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.cli.Main$LaunchCommand.call(Main.java:461) ~[org.apache.brooklyn-brooklyn-cli-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.cli.Main$LaunchCommand.call(Main.java:196) ~[org.apache.brooklyn-brooklyn-cli-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.cli.AbstractMain.execCli(AbstractMain.java:252) [org.apache.brooklyn-brooklyn-cli-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.cli.AbstractMain.execCli(AbstractMain.java:244) [org.apache.brooklyn-brooklyn-cli-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]
        at org.apache.brooklyn.cli.Main.main(Main.java:123) [org.apache.brooklyn-brooklyn-cli-0.10.0-SNAPSHOT.jar:0.10.0-SNAPSHOT]

This means something must have put/removed to the BrooklynProperties while the main thread is still starting. This is done, for example, by BrooklynCampPlatform.setConfigKeyAtManagmentContext(). It is also done in brooklyn-tosca. There are probably other places as well.

The javadoc of BrooklynPropertiesImpl says:

 * TODO methods in this class are not thread safe.
 * intention is that they are set during startup and not modified thereafter.

It seems to me the best thing is to make BrooklynPropertiesImpl thread safe. Also I'd prefer if the impl didn't just extend LinkedHashMap. That gives up a lot of control over how we implement it (e.g. makes it fiddly to make it thread-safe).