We decided to start experimenting with Standby Replicas of our State Stores by setting the following configuration setting:
Most applications did okay with this except for one that used an in memory state store instead of a persistent state store. With the new configuration, the first instance of this application booted fine. When the second instance came up, both instances crashed with the following exception:
Monit attempted to restart both instances but they would just continue to crash over and over again. The state store in our problematic application is declared like so:
Luckily we had a config switch in place that could turn on an alternate, persistent state store. As soon as we flipped to the persistent state store, things started working as we expected.