[CASSANDRA-10969] long-running cluster sees bad gossip generation when a node restarts

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.13, 2.2.5, 3.0.3, 3.3
• Component/s: Legacy/Coordination

Environment

4-node Cassandra 2.1.1 cluster, each node running on a Linux 2.6.32-431.20.3.dl6.x86_64 VM

• Severity: Normal
• Since Version: 2.1.1
• Assignee: Joel Knighton
• Reporter: T. David Hudson

Authors

Joel Knighton

Reviewers

Stefania Alborghetti

• Created: 05/Jan/16 20:01
• Updated: 16/Apr/19 09:30
• Resolved: 22/Jan/16 14:46

Description

One of the nodes in a long-running Cassandra 2.1.1 cluster (not under my control) restarted. The remaining nodes are logging errors like this:
"received an invalid gossip generation for peer xxx.xxx.xxx.xxx; local generation = 1414613355, received generation = 1450978722"

The gap between the local and received generation numbers exceeds the one-year threshold added for CASSANDRA-8113. The system clocks are up-to-date for all nodes.

If this is a bug, the latest released Gossiper.java code in 2.1.x, 2.2.x, and 3.0.x seems not to have changed the behavior that I'm seeing.

I presume that restarting the remaining nodes will clear up the problem, whence the minor priority.