Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-14881

Inconsistent behaviour of counter update during cassandra upgrade 2.1.16 -> 3.11.2

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Normal
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: Legacy/Core
    • Labels:
      None
    • Environment:
      // code placeholder
      
    • Severity:
      Normal
    • Since Version:

      Description

      I have 3 node (A,B,C) 2.1.16 cassandra cluster which i am upgrading to cassandra 3.11.2.
      My current cluster status is node a has been upgrade to 3.11.2, B is down, and C is on cassandra 2.1.16

      when i run counter update using cqlsh it is behaving strange inconsistent way , sometimes the update actually applied and sometimes it does not apply at all.

      See the below example of cqlsh logged in to node A:
      ===============Incorrect update : update not applied ========

      user@cqlsh:ks> select * from "counterTable";
      
       key              | column1        | value
      ------------------+----------------+-------
       key1 | count |     0
       
      user@cqlsh:ks> UPDATE "counterTable" SET value = value + 100 WHERE key = 'key1' AND column1 = 'count';
      
      user@cqlsh:ks> select * from "counterTable";
      
       key              | column1        | value
      ------------------+----------------+-------
       key1 | count |     0

       ===============Correct update : update applied successfully ========

      user@cqlsh> USE ks ;
      user@cqlsh:ks> select * from "counterTable";
      
       key              | column1        | value
      ------------------+----------------+-------
       key1 | count |  -100
       
      user@cqlsh:ks> UPDATE "counterTable" SET value = value + 100 WHERE key = 'key1' AND column1 = 'count';
      
      user@cqlsh:ks> select * from "counterTable";
      
       key              | column1        | value
      ------------------+----------------+-------
       key1 | count |     0

      I have attached the output with trace enabled (with actual timestamp) for both correct and incorrect counter update .

      What is the reason of this weird behavior ?

      Below is my table details:

      CREATE TABLE ks."counterTable" (
          key text,
          column1 text,
          value counter,
          PRIMARY KEY (key, column1)
      ) WITH COMPACT STORAGE
          AND CLUSTERING ORDER BY (column1 ASC)
          AND bloom_filter_fp_chance = 0.01
          AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
          AND comment = ''
          AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
          AND compression = {'chunk_length_in_kb': '64', 'class': 'org.apache.cassandra.io.compress.LZ4Compressor'}
          AND crc_check_chance = 1.0
          AND dclocal_read_repair_chance = 0.1
          AND default_time_to_live = 0
          AND gc_grace_seconds = 864000
          AND max_index_interval = 2048
          AND memtable_flush_period_in_ms = 0
          AND min_index_interval = 128
          AND read_repair_chance = 0.0
          AND speculative_retry = '99PERCENTILE';

      Note : When all nodes are successfully upgraded OR when 2 nodes gets upgraded to 3.11.2 and 3rd non-upgraded node is down then we don't see such issue . couner update works as expected

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                laxmikant99 Laxmikant Upadhyay
              • Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: