Low Description
Set up a cluster of two nodes A and B.
Disconnect the network between them.
Wait for nodetool status on both nodes to show the other node as down.
On A do nodetool removenode B.
Wait a little until:
On A nodetool status only shows A.
On B nodetool status still shows A and B with A down.
Reconnect the network.
Wait a little.
Check nodetool status on both nodes.
What one actually sees:
On A nodetool status still only shows A.
On B nodetool status shows A and B with both up.
So B can see A as up, but A can't see B as up.. That is inconsistent.
What we expect to see:
On A nodetool status shows A and B with both up.
On B nodetool status shows A and B with both up.
Restart Cassandra on either node and then one will get the expected output.
Does not seem to matter which node is the seed.
We have an automated script to removenode nodes that are down for a period of time. So if a node was isolated from the others for a time, it could get removed by the others and the cluster would not reform when the network was reconnected. So though unlikely, it is not impossible for this to happen in practice.