[CASSANDRA-11102] Data lost during compaction

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 3.0.3, 3.3
• Component/s: Local/Compaction

Environment

Cassandra 3.2.1 (single node, 5 node cluster)
JDK 8

• Severity: Critical
• Assignee: Sylvain Lebresne
• Reporter: Jaroslav Kamenik

Authors

Sylvain Lebresne

Reviewers

Marcus Eriksson

• Created: 01/Feb/16 14:10
• Updated: 16/Apr/19 09:30
• Resolved: 02/Feb/16 13:35

Description

We have experienced data loses in some tables during few weeks since update to cassandra 3.0. I thing I successfully found test case now.

Step one - test table:

CREATE TABLE aaa (
r int,
c1 int,
c2 ascii,
PRIMARY KEY (r, c1, c2));

Step two - run few queries:

insert into aaa (r, c1, c2) values (1,2,'A');
delete from aaa where r=1 and c1=2 and c2='B';
insert into aaa (r, c1, c2) values (2,3,'A');
delete from aaa where r=2 and c1=3 and c2='B';
insert into aaa (r, c1, c2) values (3,4,'A');
delete from aaa where r=3 and c1=4 and c2='B';
insert into aaa (r, c1, c2) values (4,5,'A');
delete from aaa where r=4 and c1=5 and c2='B';

It creates 4 rows (select count says 4) and 4 tombstones.

Step 3 - Restart Cassandra

You will see new files written into C* data folder. I tried sstable-tools to print table structure, it shows 4 rows, data and tombstones are there.

Step 4 - set GC grace to 1 to force tombstone removing during compaction.

alter table aaa with GC_GRACE_SECONDS = 1;

Step 5 - Compact tables

./nodetool compact

aaa files dissapeares during compaction.
select count says 0
compaction history says
... aaa 2016-02-01T14:24:01.433 329 0 {}

Issue Links

• is duplicated by: CASSANDRA-11068 Entire row is compacted away if remaining cells are tombstones expiring after gc_grace_seconds