[CASSANDRA-12144] Undeletable / duplicate rows after upgrading from 2.2.4 to 3.0.7

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.9, 3.8
• Component/s: Local/SSTable
• Severity: Normal
• Assignee: Alex Petrov
• Reporter: Stanislav Vishnevskiy

Authors

Alex Petrov

Reviewers

Sylvain Lebresne

• Created: 06/Jul/16 11:07
• Updated: 20/Apr/20 18:02
• Resolved: 19/Jul/16 10:08

Description

We upgraded our cluster today and now have a some rows that refuse to delete.

Here are some example traces.

https://gist.github.com/vishnevskiy/36aa18c468344ea22d14f9fb9b99171d

Even weirder.

Updating the row and querying it back results in 2 rows even though the id is the clustering key.

user_id            | id                 | since                    | type
-------------------+--------------------+--------------------------+------
116138050710536192 | 153047019424972800 |                     null |    0
116138050710536192 | 153047019424972800 | 2016-05-30 14:53:08+0000 |    2

And then deleting it again only removes the new one.

cqlsh:discord_relationships> DELETE FROM relationships WHERE user_id = 116138050710536192 AND id = 153047019424972800;
cqlsh:discord_relationships> SELECT * FROM relationships WHERE user_id = 116138050710536192 AND id = 153047019424972800;

 user_id            | id                 | since                    | type
--------------------+--------------------+--------------------------+------
 116138050710536192 | 153047019424972800 | 2016-05-30 14:53:08+0000 |    2

We tried repairing, compacting, scrubbing. No Luck.

Not sure what to do. Is anyone aware of this?

Issue Links

• duplicates: CASSANDRA-12246 Cassandra v2.2 to v3.0.9 upgrade failed
• is duplicated by: CASSANDRA-11887 Duplicate rows after a 2.2.5 to 3.0.4 migration
• is duplicated by: CASSANDRA-12756 Duplicate (cql)rows for the same primary key
• is related to: CASSANDRA-13125 Duplicate rows after upgrading from 2.1.16 to 3.0.10/3.9
• is superceded by: CASSANDRA-14008 RTs at index boundaries in 2.x sstables can create unexpected CQL row in 3.x