[CASSANDRA-15778] CorruptSSTableException after a 2.1 SSTable is upgraded to 3.0, failing reads

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.21
• Component/s: Local/Compaction, Local/SSTable
• Bug Category: Correctness - Unrecoverable Corruption / Loss
• Severity: Critical
• Complexity: Challenging
• Discovered By: User Report
• Platform: All
• Since Version: 3.0 alpha 1

Source Control Link

https://github.com/apache/cassandra/commit/c4064dd80e427aec7c04e8e2e1e4630d6c8087b6

Test and Documentation Plan

Test added; manually tested additional scenarios of upgrade to 3.11, dropping compact storage, and reading in 4.0. However, this is going to be fully handled in scope of CASSANDRA-15811.

• Assignee: Alex Petrov
• Reporter: Sumanth Pasupuleti

Authors

Alex Petrov

Reviewers

Alex Petrov, David Capwell, Sylvain Lebresne

• Created: 01/May/20 15:39
• Updated: 01/Aug/21 12:18
• Resolved: 04/Jun/20 18:21

Description

Below is the exception with stack trace. This issue is consistently reproduce-able.

ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]ERROR [SharedPool-Worker-1] 2020-05-01 14:57:57,661 AbstractLocalAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /mnt/data/cassandra/data/<ks>/<cf-fda511301fb311e7bd79fd24f2fcfb0d/md-10151-big-Data.db at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:349) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator.hasNext(AbstractSSTableIterator.java:220) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.columniterator.SSTableIterator.hasNext(SSTableIterator.java:33) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:131) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:87) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.rows.UnfilteredRowIteratorSerializer.serialize(UnfilteredRowIteratorSerializer.java:77) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$Serializer.serialize(UnfilteredPartitionIterators.java:294) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.build(ReadResponse.java:187) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:180) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.ReadResponse$LocalDataResponse.<init>(ReadResponse.java:176) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.ReadResponse.createDataResponse(ReadResponse.java:76) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.ReadCommand.createResponse(ReadCommand.java:341) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:47) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_231] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$FutureTask.run(AbstractLocalAwareExecutorService.java:165) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.concurrent.AbstractLocalAwareExecutorService$LocalSessionFutureTask.run(AbstractLocalAwareExecutorService.java:137) [nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:109) [nf-cassandra-3.0.19.8.jar:3.0.19.8] at java.lang.Thread.run(Thread.java:748) [na:1.8.0_231]Caused by: java.lang.ArrayIndexOutOfBoundsException: 121 at org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:425) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:170) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:151) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:140) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:172) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:336) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] ... 27 common frames omitted

Caused by: java.lang.ArrayIndexOutOfBoundsException: 121
    at org.apache.cassandra.db.ClusteringPrefix$Deserializer.prepare(ClusteringPrefix.java:425) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
    at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.prepareNext(UnfilteredDeserializer.java:170) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
    at org.apache.cassandra.db.UnfilteredDeserializer$CurrentDeserializer.hasNext(UnfilteredDeserializer.java:151) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
    at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.computeNext(SSTableIterator.java:140) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
    at org.apache.cassandra.db.columniterator.SSTableIterator$ForwardReader.hasNextInternal(SSTableIterator.java:172) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
    at org.apache.cassandra.db.columniterator.AbstractSSTableIterator$Reader.hasNext(AbstractSSTableIterator.java:336) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8] ... 27 common frames omitted

Column family definition

CREATE TABLE <keyspace>."<cf>" (
 key text,
 value text,
 PRIMARY KEY (key, value)
 ) WITH COMPACT STORAGE
 AND CLUSTERING ORDER BY (value ASC)
 AND bloom_filter_fp_chance = 0.01
 AND caching = {'keys': 'ALL', 'rows_per_partition': 'NONE'}
 AND comment = ''
 AND compaction = {'class': 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', 'max_threshold': '32', 'min_threshold': '4'}
 AND compression = {'enabled': 'false'}
 AND crc_check_chance = 1.0
 AND dclocal_read_repair_chance = 0.1
 AND default_time_to_live = 0
 AND gc_grace_seconds = 864000
 AND max_index_interval = 2048
 AND memtable_flush_period_in_ms = 0
 AND min_index_interval = 128
 AND read_repair_chance = 0.0
 AND speculative_retry = '99PERCENTILE';

Issue Links

• is related to: CASSANDRA-15373 validate value sizes in LegacyLayout
• relates to: CASSANDRA-15799 CorruptSSTableException when compacting a 3.0 format sstable that was originally created as an outcome of 2.1 sstable upgrade
• relates to: CASSANDRA-15790 EmptyType doesn't override writeValue so could attempt to write bytes when expected not to