[CASSANDRA-15097] Avoid updating unchanged gossip state

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.19, 3.11.5, 4.0-alpha1, 4.0
• Component/s: Cluster/Gossip
• Bug Category: Degradation - Performance Bug/Regression
• Severity: Low
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.0

Source Control Link

3f70e7c72c703bc323b169a28e8754ce67d4e479

Test and Documentation Plan

Unittest is passed. And the code is committed and running in Instagram production environment.

• Assignee: Jay Zhuang
• Reporter: Jay Zhuang

Authors

Jay Zhuang

Reviewers

Sam Tunnicliffe

• Created: 23/Apr/19 22:12
• Updated: 15/May/20 08:00
• Resolved: 19/Jul/19 04:50

Description

The node might get unchanged gossip states, the state might be just updated after sending a GOSSIP_SYN, then it will get the state that is already up to date. If the heartbeat in the GOSSIP_ACK message is updated, it will unnecessary re-apply the same state again, which could be costly like updating token change.
It's very likely to happen for large cluster when a node startup, as the first gossip message will sync all endpoints tokens, it could take some time (in our case about 200 seconds), during that time, it keeps gossip with other node and get the full token states. Which causes lots of pending gossip tasks.