[CASSANDRA-13441] Schema version changes for each upgraded node in a rolling upgrade, causing migration storms

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.14, 3.11.0, 4.0-alpha1, 4.0
• Component/s: Legacy/Distributed Metadata
• Bug Category: Degradation - Performance Bug/Regression
• Severity: Normal
• Assignee: Jeff Jirsa
• Reporter: Jeff Jirsa

Authors

Jeff Jirsa

Reviewers

Aleksey Yeschenko

• Created: 12/Apr/17 18:23
• Updated: 15/May/20 08:02
• Resolved: 24/Apr/17 16:13

Description

In versions < 3.0, during a rolling upgrade (say 2.0 -> 2.1), the first node to upgrade to 2.1 would add the new tables, setting the new 2.1 version ID, and subsequently upgraded hosts would settle on that version.

When a 3.0 node upgrades and writes its own new-in-3.0 system tables, it'll write the same tables that exist in the schema with brand new timestamps. As written, this will cause all nodes in the cluster to change schema (to the version with the newest timestamp). On a sufficiently large cluster with a non-trivial schema, this could cause (literally) millions of migration tasks to needlessly bounce across the cluster.

Issue Links

• is duplicated by: CASSANDRA-11983 Migration task failed to complete
• relates to: CASSANDRA-13812 Missing system keyspace tables are not created