Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15433

Pending ranges are not recalculated on keyspace creation

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Changes Suggested
    • Priority: Normal
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: Cluster/Membership
    • Labels:
      None
    • Bug Category:
      Correctness - Recoverable Corruption / Loss
    • Severity:
      Normal
    • Complexity:
      Normal
    • Discovered By:
      User Report
    • Platform:
      All
    • Impacts:
      None
    • Since Version:
    • Test and Documentation Plan:
      Hide

      in-jvm dtest

       

      Show
      in-jvm dtest  

      Description

      When a node begins bootstrapping, Cassandra recalculates pending tokens for each keyspace that exists when the state change is observed (in StorageService:handleState*). When new keyspaces are created, we do not recalculate pending ranges (around Schema:merge). As a result, writes for new keyspaces are not received by nodes in BOOT or BOOT_REPLACE modes. When bootstrapping finishes, the node which just bootstrapped will not have data for the newly created keyspace.

      Consider a ring with bootstrapped nodes A, B, and C. Node D is pending, and when it finishes bootstrapping, C will cede ownership of some ranges to D. A quorum write is acknowledged by C and A. B missed the write, and the coordinator didn't send it to D at all. When D finishes bootstrapping, the quorum B+D will not contain the mutation.

      Steps to reproduce:

      1. Join a node in BOOT mode
      2. Create a keyspace
      3. Send writes to that keyspace
      4. On the joining node, observe that nodetool cfstats records zero writes to the new keyspace

      I have observed this directly in Cassandra 3.0, and based on my reading the code, I believe it affects up through trunk.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                sumanth.pasupuleti Sumanth Pasupuleti
                Reporter:
                josnyder Josh Snyder
                Authors:
                Sumanth Pasupuleti
                Reviewers:
                Alex Petrov
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated: