Uploaded image for project: 'Apache Cassandra'
  1. Apache Cassandra
  2. CASSANDRA-9022

Node Cleanup deletes all its data after a new node joined the cluster

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Urgent
    • Resolution: Fixed
    • 2.1.5
    • None
    • Critical

    Description

      I try to add a node in my cluster and doing some cleanup deleted all my data on a node. This makes the cluster totally broken since all next read seem to not be able to validate the data. Even a repair on the problematic node doesn't fix the issue. I've attached the bisect script used and the output results of the procedure.

      Procedure to reproduce:

      ccm stop && ccm remove
      ccm create -n 2 --install-dir=path/to/cassandra-2.1/branch demo
      ccm start
      ccm node1 stress -- write n=1000000 -schema replication\(factor=2\) -rate threads=50
      ccm node1 nodetool status
      ccm add -i 127.0.0.3 -j 7400 node3 # no auto-boostrap
      ccm node3 start
      ccm node1 nodetool status
      ccm node3 repair
      ccm node3 nodetool status
      ccm node1 nodetool cleanup
      ccm node2 nodetool cleanup
      ccm node3 nodetool cleanup
      ccm node1 nodetool status
      ccm node1 repair
      ccm node1 stress -- read n=1000000 ## CRASH Data returned was not validated ?!?
      

      bisec script output:

      $ git bisect start cassandra-2.1 cassandra-2.1.3
      $ git bisect run ~/dev/cstar/cleanup_issue/bisect.sh
      ...
      4b05b204acfa60ecad5672c7e6068eb47b21397a is the first bad commit
      commit 4b05b204acfa60ecad5672c7e6068eb47b21397a
      Author: Benedict Elliott Smith <benedict@apache.org>
      Date:   Wed Feb 11 15:49:43 2015 +0000
      
          Enforce SSTableReader.first/last
          
          patch by benedict; reviewed by yukim for CASSANDRA-8744
      
      :100644 100644 3f0463731e624cbe273dcb3951b2055fa5d9e1a2 b2f894eb22b9102d410f1eabeb3e11d26727fbd3 M      CHANGES.txt
      :040000 040000 51ac2a6cd39bd2377c2e1ed6693ef789ab65a26c 79fa2501f4155a64dca2bbdcc9e578008e4e425a M      src
      bisect run success
      

      Attachments

        1. 9022.txt
          7 kB
          Benedict Elliott Smith
        2. bisect.sh
          0.6 kB
          Alan Boudreault
        3. results_cassandra_2.1_branch.txt
          3 kB
          Alan Boudreault
        4. results_cassandra_2.1.3.txt
          2 kB
          Alan Boudreault

        Activity

          People

            benedict Benedict Elliott Smith
            aboudreault Alan Boudreault
            Benedict Elliott Smith
            Sylvain Lebresne
            Alan Boudreault Alan Boudreault
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: