Mutable metadata has been a very rich source of bugs. There are multiple processing modifying for rebuilding ensembles, closing, fencing, autorecovery etc.
The local metadata should be a mirror of the metadata in zookeeper. So we should only modify the local metadata as a whole. The first part of this is making it immutable, so that it can only be modified with a builder.