[CASSANDRA-7245] Out-of-Order keys with stress + CQL3

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1 rc2
• Severity: Normal
• Assignee: T Jake Luciani
• Reporter: Pavel Yaskevich

Authors

T Jake Luciani

Reviewers

Benedict Elliott Smith

• Created: 16/May/14 01:13
• Updated: 16/Apr/19 09:31
• Resolved: 05/Jun/14 18:24

Description

We have been generating data (stress with CQL3 prepared) for CASSANDRA-4718 and found following problem almost in every SSTable generated (~200 GB of data and 821 SSTables).

We set up they keys to be 10 bytes in size (default) and population between 1 and 600000000.

Once I ran 'sstablekeys' on the generated SSTable files I got following exceptions:

There is a problem with sorting of normal looking keys:

30303039443538353645
30303039443745364242
java.io.IOException: Key out of order! DecoratedKey(-217680888487824985, 30303039443745364242) > DecoratedKey(-1767746583617597213, 30303039443437454333)

00000a30303033343933
37344400001388343933
java.io.IOException: Key out of order! DecoratedKey(5440473860101999581, 37344400001388343933) > DecoratedKey(-7565486415339257200, 30303033344639443137)

30303033354244363031
30303033354133423742
java.io.IOException: Key out of order! DecoratedKey(2687072396429900180, 30303033354133423742) > DecoratedKey(-7838239767410066684, 30303033354145344534)

30303034313442354137
30343136353633340000
java.io.IOException: Key out of order! DecoratedKey(1516003874415400462, 30343136353633340000) > DecoratedKey(-9106177395653818217, 30303034313333444238)

30303035373044373435
30303035373044334631
java.io.IOException: Key out of order! DecoratedKey(-3645715702154616540, 30303035373044334631) > DecoratedKey(-4296696226469000945, 30303035373132364138)

And completely different ones:

30303041333745373543
7cd045c59a90d7587d8d
java.io.IOException: Key out of order! DecoratedKey(-3595402345023230196, 7cd045c59a90d7587d8d) > DecoratedKey(-5146766422778260690, 30303041333943303232)

30303033323144444144
30303033323346343932
java.io.IOException: Key out of order! DecoratedKey(7071845511166615635, 30303033323346343932) > DecoratedKey(5233296131921119414, 53d83e00000012287e03)

30303034314531374431
3806734b256c27e41ec2
java.io.IOException: Key out of order! DecoratedKey(-7720474642702543193, 3806734b256c27e41ec2) > DecoratedKey(-8072288379146044663, 30303034314136413343)

And sometimes there is no problem at all:

30303033353144463637
0000002a31b3b31a1c2f
5d616dd38211ebb5d6ec
44423645000013880000
00001388138844463744
30303033353143394343

It's worth to mention that we have got 22 timeout exceptions but number of out-of-order keys is much larger than that.