[CASSANDRA-5669] Connection thrashing in multi-region ec2 during upgrade, due to messaging version

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Fixed
• Fix Version/s: 1.2.6, 2.0 beta 1
• Labels: ec2 ec2multiregionsnitch gossip
• Severity: Low
• Assignee: Jason Brown
• Reporter: Jason Brown

Authors

Jason Brown

Reviewers

Jonathan Ellis

• Created: 19/Jun/13 22:51
• Updated: 16/Apr/19 09:32
• Resolved: 20/Jun/13 19:24

Description

While debugging the upgrading scenario described in CASSANDRA-5660, I discovered the ITC.close() will reset the message protocol version of a peer node that disconnects. CASSANDRA-5660 has a full description of the upgrade path, but basically the Ec2MultiRegionSnitch will close connections on the publicIP addr to reconnect on the privateIp, and this causes ITC to drop the message protocol version of previously known nodes. I think we want to hang onto that version so that when the newer node (re-)connects to the lower node version, it passes the correct protocol version rather than the current version (too high for the older node),the connection attempt getting dropped, and going through the dance again.

To clarify, the 'thrashing' is at a rather low volume, from what I observed. Anecdotaly, perhaps one connection per second gets turned over.