Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15918

materialized view rebuild automatically after drop multiple views

    XMLWordPrintableJSON

    Details

    • Platform:
      All
    • Impacts:
      None

      Description

      Background:

      Cassandra version: 3.0.12

      Our cassandra cluster has 9 host for DC1 and 3 host for DC2,

      each host :

      node memory disk
      DC1 1 256 GB 5*788GB
      DC1 2 256 GB 5*788GB
      DC1 3 256 GB 5*788GB
      DC1 4 256 GB 5*788GB
      DC1 5 256 GB 5*788GB
      DC1 6 256 GB 5*788GB
      DC1 7 512 GB 5 TB
      DC1 8 512 GB 5 TB
      DC1 9 512 GB 5 TB
      DC2 1 256 GB 8*788GB
      DC2 2 256 GB 8*788GB
      DC2 3 256 GB 8*788GB

      by using nodetool status, node load in DC1 is about 1.5 TB, node load in DC2 is about 4 TB

      QPS: 270

      -------------------------------------

      Problem we met:

      In DC1 1 node, enter the cql command line and execute command like following in sametime:

      "drop materialized view if exists view1; 

      drop materialized view if exists view2;

      drop materialized view if exists view3;

      drop materialized view if exists view4;"

      after a while, command line display warning like "schema version mismatch detected..." (sorry we cannot find the exact output for that time)

      After that we find view files in node: "DC1 7" hasn't be deleted yet.

      at this moment, we find performance of cluster drop sharp, the cluster almost stop response to any request.

      by runing:  select * from system.views_builds_in_progress;

      we can see several views were building.

      then we execte:

      1,   nodetool stop VIEW_BUILD  in each node

      2, in cql: delete from system.views_builds_in_progress where view_name=XXXX

      3, rolling restart cassandra nodes

       

      about an hours later, performance increase to normal.

      --------------------------------------------------------------

      Why this happen?

      How to avoid this problem?

      Any better way to deal with this problem?

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              mark404 chonghao li
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: