[CASSANDRA-14013] Keyspace named "snapshots" is empty after service restart

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0.8, 4.1.1, 5.0-alpha1, 5.0
• Component/s: Legacy/Core, Local/Snapshots
• Severity: Normal
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/6e6846892a5d7ec2feaf2c35d380ed5975d09517

Test and Documentation Plan

CI and added tests

• Assignee: Stefan Miklosovic
• Reporter: Gregor Uhlenheuer

Authors

Stefan Miklosovic

Reviewers

Paulo Motta

• Created: 13/Nov/17 16:30
• Updated: 12/Sep/23 13:03
• Resolved: 17/Jan/23 03:55

Description

I am posting this bug in hope to discover the stupid mistake I am doing because I can't imagine a reasonable answer for the behavior I see right now

In short words, I do observe data loss in a keyspace called snapshots after restarting the Cassandra service. Say I do have 1000 records in a table called snapshots.test_idx then after restart the table has less entries or is even empty.

My kind of "mysterious" observation is that it happens only in a keyspace called snapshots...

Steps to reproduce

These steps to reproduce show the described behavior in "most" attempts (not every single time though).

# create keyspace
CREATE KEYSPACE snapshots WITH replication = {'class': 'SimpleStrategy', 'replication_factor': 1};

# create table
CREATE TABLE snapshots.test_idx (key text, seqno bigint, primary key(key));

# insert some test data
INSERT INTO snapshots.test_idx (key,seqno) values ('key1', 1);
...
INSERT INTO snapshots.test_idx (key,seqno) values ('key1000', 1000);

# count entries
SELECT count(*) FROM snapshots.test_idx;
1000

# restart service
kill <cassandra-pid>
cassandra -f

# count entries
SELECT count(*) FROM snapshots.test_idx;
0

I hope someone can point me to the obvious mistake I am doing

This happened to me using both Cassandra 3.9 and 3.11.0

Issue Links

• is related to: CASSANDRA-16235 SSTableLoader issue when restoring a table named backups
• is related to: CASSANDRA-16251 SSTableLoader documentation needs improvement
• relates to: CASSANDRA-18168 Serialize sstable keyspace and table metadata on StatsMetadata component