[CASSANDRA-19623] Read fail with "illegal RT bounds sequence" after migrating data from 2.2.19 to 3.0.30

• Type: Bug
• Status: Open
• Priority: Normal
• Resolution: Unresolved
• Fix Version/s: 3.0.x
• Component/s: Legacy/Local Write-Read Paths
• Bug Category: Availability - Unavailable
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Assignee: Unassigned
• Reporter: Klay
• Created: 08/May/24 01:06
• Updated: 09/May/24 16:13

Description

After migrating data from 2.2.19 to 3.0.30, reading the legacy data in 3.0.30 would fail with the following exception.

INFO  [main] 2024-05-08 00:42:42,397 CassandraDaemon.java:653 - Startup complete
ERROR [SharedPool-Worker-2] 2024-05-08 00:42:49,975 AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread Thread[SharedPool-Worker-2,10,main]
java.lang.RuntimeException: java.lang.IllegalStateException: SSTABLE UnfilteredRowIterator for ks.tb has an illegal RT bounds sequence: unexpected end bound or boundary Marker INCL_START_BOUND(1)@1715128933051779/1715128933
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2656)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalStateException: SSTABLE UnfilteredRowIterator for ks.tb has an illegal RT bounds sequence: unexpected end bound or boundary Marker INCL_START_BOUND(1)@1715128933051779/1715128933
        at org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120)
        at org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.applyToMarker(RTBoundValidator.java:87)
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:144)
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
        at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:297)
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180)
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176)
        at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
        at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:347)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1914)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2652)
        ... 5 common frames omitted 

Reproduce

1. Start up cassandra-2.2.19, single node. Set column_index_size_in_kb in cassandra.yaml to 1.

// cassandra.yaml
column_index_size_in_kb: 1 

2. Execute the following commands (I masked out all data in the INSERT command using a long string of 'A' to make it simpler)

CREATE KEYSPACE  ks WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor' : 1 };
CREATE TABLE  ks.tb (c3 INT, c7 INT, c0 TEXT,c6 set<TEXT>, PRIMARY KEY (c3, c7, c0));
DELETE FROM ks.tb WHERE c3 = 1 AND c7 = 1;
DELETE FROM ks.tb WHERE c3 = 1 AND c7 = 1 AND c0 = 'e';
INSERT INTO ks.tb (c7, c3, c6, c0) VALUES (1,1,{'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA','AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA','AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'},'AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA'); 

Perform a read in the old version, it works normally

cqlsh> SELECT c7, c0, c3 FROM ks.tb WHERE c3 = 1 AND c7 = 1 ORDER BY c7 DESC;

 c7 | c0                                               | c3
----+--------------------------------------------------+----
  1 | AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA |  1

(1 rows) 

3. Drain and stop 2.2.19.

bin/nodetool -h ::FFFF:127.0.0.1 drain
bin/nodetool -h ::FFFF:127.0.0.1 stopdaemon; 

4. Copy data into 3.0.30 and start up (using default config is okay), read the legacy data with the following read command

SELECT c7, c0, c3 FROM ks.tb WHERE c3 = 1 AND c7 = 1 ORDER BY c7 DESC; 

It would fail and throw an error in system.log.

➜  apache-cassandra-3.0.30 bin/cqlsh
Connected to Test Cluster at 127.0.0.1:9042.
[cqlsh 5.0.1 | Cassandra 3.0.30-SNAPSHOT | CQL spec 3.4.0 | Native protocol v4]
Use HELP for help.
cqlsh> SELECT c7, c0, c3 FROM ks.tb WHERE c3 = 1 AND c7 = 1 ORDER BY c7 DESC;
ReadFailure: Error from server: code=1300 [Replica(s) failed to execute read] message="Operation failed - received 0 responses and 1 failures" info={'failures': 1, 'received_responses': 0, 'required_responses': 1, 'consistency': 'ONE'} 

system.log

INFO  [main] 2024-05-08 00:52:11,649 CassandraDaemon.java:653 - Startup complete
ERROR [SharedPool-Worker-2] 2024-05-08 00:52:13,695 AbstractLocalAwareExecutorService.java:166 - Uncaught exception on thread Thread[SharedPool-Worker-2,10,main]
java.lang.RuntimeException: java.lang.IllegalStateException: SSTABLE UnfilteredRowIterator for ks.tb has an illegal RT bounds sequence: unexpected end bound or boundary Marker INCL_START_BOUND(1)@1715129489466439/1715129489
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2656)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:162)
        at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:134)
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109)
        at java.lang.Thread.run(Thread.java:750)
Caused by: java.lang.IllegalStateException: SSTABLE UnfilteredRowIterator for ks.tb has an illegal RT bounds sequence: unexpected end bound or boundary Marker INCL_START_BOUND(1)@1715129489466439/1715129489
        at org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.ise(RTBoundValidator.java:120)
        at org.apache.cassandra.db.transform.RTBoundValidator$RowsTransformation.applyToMarker(RTBoundValidator.java:87)
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:144)
        at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87)
        at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77)
        at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:297)
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187)
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180)
        at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176)
        at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76)
        at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:347)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1914)
        at org.apache.cassandra.service 

I attached the data.tar.gz file generated from the 2.2.19, using 3.0.30 to start with it and execute the above read command can directly reproduce it.

I also attached the cassandra.yaml file used in 2.2.19.