- I did the following
- /live_nodes reflects that the node is gone.
- This is the only message which gets logged on the node1 server after killing node2
- The graph shows the node2 as 'Gone' state
- clusterstate.json keeps showing the replica as 'active'
One immediate problem I can see is that AutoAddReplicas doesn't work since the clusterstate.json never changes. There might be more features which are affected by this.
On first thought I think we can handle this - The shard leader could listen to changes on /live_nodes and if it has replicas that were on that node, mark it as 'down' in the clusterstate.json?