[CASSANDRA-6648] Race condition during node bootstrapping

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 1.2.15, 2.0.5
• Labels: qa-resolved
• Severity: Critical
• Since Version: 1.2.14
• Assignee: Sergio Bossa
• Reporter: Sergio Bossa

Authors

Sergio Bossa

Reviewers

Tom Hobbs

• Tester: Ryan McGuire
• Created: 04/Feb/14 12:37
• Updated: 16/Apr/19 09:31
• Resolved: 05/Feb/14 00:37

Description

When bootstrapping a new node, data is "missing" as if the new node didn't actually bootstrap, which I tracked down to the following scenario:

1) New node joins token ring and waits for schema to be settled before actually bootstrapping.
2) The schema scheck somewhat passes and it starts bootstrapping.
3) Bootstrapping doesn't find the ks/cf that should have received from the other node.
4) Queries at this point cause NPEs, until when later they "recover" but data is missed.

The problem seems to be caused by a race condition between the migration manager and the bootstrapper, with the former running after the latter.
I think this is supposed to protect against such scenarios:

            while (!MigrationManager.isReadyForBootstrap())
            {
                setMode(Mode.JOINING, "waiting for schema information to complete", true);
                Uninterruptibles.sleepUninterruptibly(1, TimeUnit.SECONDS);
            }

But MigrationManager.isReadyForBootstrap() implementation is quite fragile and doesn't take into account "slow" schema propagation.

Issue Links

• is duplicated by: CASSANDRA-6650 Bootstrapped nodes don't know about existing keyspaces