The snitch can be set via StorageService over JMX, for what reason I’m unsure. If this were to happen, we might encounter the following problems:
- If the effective local DC were to change, we would not update it. Perhaps changing the local DC of a node should be rejected and cause it to fail, but presently, it would simply result in our disagreeing with the snitch.
- During the transition, routing of queries might be broken, as we fetch the snitch multiple times in different locations when deciding where to route our query and writes. It’s not clear what the outcome of a discordant view of the ring would be.
Probably, changing this information in a live cluster is dangerous and we should actually reject any effective changes to rack, or DC for any node. But presently we don’t seem to corroborate that this information remains the same. We don’t seem to perform any cluster wide confirmation that this data is consistent, generally, which perhaps we should also consider.