[CASSANDRA-5898] nodetool cleanup interacting badly with LCS

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Won't Fix
• Fix Version/s: 1.2.0
• Labels: lcs
• Severity: Normal
• Assignee: Yuki Morishita
• Reporter: Jeremiah Jordan

Authors

Yuki Morishita

• Created: 16/Aug/13 19:51
• Updated: 16/Apr/19 09:32
• Resolved: 19/Aug/13 20:48

Description

Then run cassandra-stress read, cassandra-stress write and "nodetool cleanup" all at the same time, a few times, it can cause overlapping row errors.

ERROR [CompactionExecutor:3] 2013-08-16 19:20:26,174 LeveledManifest.java (line 223) At level 1, SSTableReader(path='/raid0/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-245-Data.db') [DecoratedKey(204853724659241194183955214890519, 30303132343830), DecoratedKey(69227335985728660912035125310473966323, 30393537373332)] overlaps SSTableReader(path='/raid0/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-243-Data.db') [DecoratedKey(217896711032704014921095870827202, 30333635363932), DecoratedKey(71430242198281555888954138354238066233, 30333035343132)].  This is caused by a bug in Cassandra 1.1.0 .. 1.1.3.  Sending back to L0.  If you have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable
ERROR [CompactionExecutor:3] 2013-08-16 19:20:26,174 LeveledManifest.java (line 223) At level 1, SSTableReader(path='/raid0/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-213-Data.db') [DecoratedKey(69231122848869715605691825585281602523, 30303236343834), DecoratedKey(138494249350583165433071928358975719316, 30393733303134)] overlaps SSTableReader(path='/raid0/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-247-Data.db') [DecoratedKey(71430378028108400200608251408083114019, 30383133313937), DecoratedKey(142788557025615975974639291059795386925, 30353437373934)].  This is caused by a bug in Cassandra 1.1.0 .. 1.1.3.  Sending back to L0.  If you have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable
ERROR [CompactionExecutor:3] 2013-08-16 19:20:26,175 LeveledManifest.java (line 223) At level 1, SSTableReader(path='/raid0/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-249-Data.db') [DecoratedKey(138495280291659088173025258556445559667, 30313039353935), DecoratedKey(170139918483433930784261689358244392849, 30313336303439)] overlaps SSTableReader(path='/raid0/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-252-Data.db') [DecoratedKey(142788613459989682219570222310727294191, 30383031333839), DecoratedKey(170140890916069420209456014750832266661, 30333731303535)].  This is caused by a bug in Cassandra 1.1.0 .. 1.1.3.  Sending back to L0.  If you have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable
ERROR [CompactionExecutor:3] 2013-08-16 19:22:28,912 LeveledManifest.java (line 223) At level 1, SSTableReader(path='/raid0/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-271-Data.db') [DecoratedKey(68545052253582928820015773344685671746, 30303739313938), DecoratedKey(119320297902762425433209003370359962009, 30373137323238)] overlaps SSTableReader(path='/raid0/cassandra/data/Keyspace1/Standard1/Keyspace1-Standard1-hf-270-Data.db') [DecoratedKey(69231122848869715605691825585281602523, 30303236343834), DecoratedKey(138494249350583165433071928358975719316, 30393733303134)].  This is caused by a bug in Cassandra 1.1.0 .. 1.1.3.  Sending back to L0.  If you have not yet run scrub, you should do so since you may also have rows out-of-order within an sstable

Steps to repro:

1. Run cassandra-stress -Z LeveledCompactionStrategy
2. Run nodetool cleanup
3. While cleanup is still running, run cassandra-stress -o write and cassandra-stress -o read at the same time.