[CASSANDRA-7734] Schema pushes (seemingly) randomly not happening

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.11, 2.1.1
• Severity: Normal
• Assignee: Aleksey Yeschenko
• Reporter: graham sanderson

Authors

Aleksey Yeschenko

Reviewers

Marcus Eriksson

• Created: 10/Aug/14 19:37
• Updated: 16/Apr/19 09:31
• Resolved: 07/Oct/14 15:46

Description

We have been seeing problems since upgrade to 2.0.9 from 2.0.5.

Basically after a while, new schema changes (we periodically add tables) start propagating very slowly to some nodes and fast to others. It looks from the logs and trace that in this case the "push" of the schema never happens (note a node has decided not to push to another node, it doesn't seem to start again) from the originating node to some of the other nodes. In this case though, we do see the other node end up pulling the schema some time later when it notices its schema is out of date.

Here is code from 2.0.9 MigrationManager.announce

       for (InetAddress endpoint : Gossiper.instance.getLiveMembers())
        {
            // only push schema to nodes with known and equal versions
            if (!endpoint.equals(FBUtilities.getBroadcastAddress()) &&
                    MessagingService.instance().knowsVersion(endpoint) &&
                    MessagingService.instance().getRawVersion(endpoint) == MessagingService.current_version)
                pushSchemaMutation(endpoint, schema);
        }

and from 2.0.5

        for (InetAddress endpoint : Gossiper.instance.getLiveMembers())
        {
            if (endpoint.equals(FBUtilities.getBroadcastAddress()))
                continue; // we've dealt with localhost already

            // don't send schema to the nodes with the versions older than current major
            if (MessagingService.instance().getVersion(endpoint) < MessagingService.current_version)
                continue;

            pushSchemaMutation(endpoint, schema);
	}

the old getVersion() call would return MessagingService.current_version if the version was unknown, so the push would occur in this case. I don't have logging to prove this, but have strong suspicion that the version may end up null in some cases (which would have allowed schema propagation in 2.0.5, but not by somewhere after that and <= 2.0.9)

Issue Links

• is duplicated by: CASSANDRA-7406 Reset version when closing incoming socket in IncomingTcpConnection should be done atomically
• is related to: CASSANDRA-8057 Record the real messaging version in all cases in OutboundTcpConnection
• relates to: CASSANDRA-6038 Support schema changes in mixed version clusters
• relates to: CASSANDRA-2860 Versioning works *too* well
• relates to: CASSANDRA-3166 Rolling upgrades from 0.7 to 0.8 not possible
• relates to: CASSANDRA-6095 INSERT query adds new value to collection type
• relates to: CASSANDRA-6700 Use real node messaging versions for schema exchange decisions