Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-10250

Executing lots of schema alters concurrently can lead to dropped alters

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Severity:
      Normal
    • Since Version:

      Description

      A recently added dtest has been flapping on cassci and has exposed an issue with running lots of schema alterations concurrently. The failures occur on healthy clusters but seem to occur at higher rates when 1 node is down during the alters.

      The test executes the following – 440 total commands:

      • Create 20 new tables
      • Drop 7 columns one at time across 20 tables
      • Add 7 columns one at time across 20 tables
      • Add one column index on each of the 7 columns on 20 tables

      Outcome is random. Majority of the failures are dropped columns still being present, but new columns and indexes have been observed to be incorrect. The logs are don’t have exceptions and the columns/indexes that are incorrect don’t seem to follow a pattern. Running a nodetool describecluster on each node shows the same schema id on all nodes.

      Attached is a python script extracted from the dtest. Running against a local 3 node cluster will reproduce the issue (with enough runs – fails ~20% on my machine).

      Also attached is the node logs from a run with when a dropped column (alter_me_7 table, column s1) is still present. Checking the system_schema tables for this case shows the s1 column in both the columns and drop_columns tables.

      This has been flapping on cassci on versions 2+ and doesn’t seem to be related to changes in 3.0. More testing needs to be done though.

      //cc Ryan McGuire

        Attachments

        1. concurrent_schema_changes.py
          3 kB
          Andrew Hust
        2. node1.log
          2.58 MB
          Andrew Hust
        3. node2.log
          2.58 MB
          Andrew Hust
        4. node3.log
          2.62 MB
          Andrew Hust

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                nutbunnies Andrew Hust
                Reviewers:
                Aleksey Yeschenko
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: