[CASSANDRA-15363] Read repair in mixed mode between 2.1 and 3.0 on COMPACT STORAGE tables causes unreadable sstables after upgrade

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.19, 3.11.5, 4.0-alpha2, 4.0
• Component/s: Consistency/Coordination
• Bug Category: Correctness - Recoverable Corruption / Loss
• Severity: Critical
• Complexity: Normal
• Discovered By: Adhoc Test
• Platform: All
• Since Version: 3.0.0

Source Control Link

https://github.com/apache/cassandra/commit/9081d0088119f02c8118598ac9fdb8ce39f0033b

Test and Documentation Plan

adds an in-jvm upgrade dtest

• Assignee: Marcus Eriksson
• Reporter: Marcus Eriksson

Authors

Marcus Eriksson

Reviewers

Aleksey Yeschenko, Benedict Elliott Smith

• Created: 18/Oct/19 09:01
• Updated: 15/May/20 08:36
• Resolved: 24/Oct/19 09:27

Description

if we have a table like this:

CREATE TABLE tbl (pk ascii, b boolean, v blob, PRIMARY KEY (pk)) WITH COMPACT STORAGE

with a cluster where node1 is 2.1 and node2 is 3.0 (during upgrade):

• node2 coordinates a delete DELETE FROM tbl WHERE pk = 'something' which node1 does not get
• node1 coordinates a quorum read SELECT * FROM tbl WHERE id = 'something' which causes a read repair
• this makes node1 flush an sstable like this:

  [
  {"key": "something",
   "metadata": {"deletionInfo": {"markedForDeleteAt":1571388944364000,"localDeletionTime":1571388944}},
   "cells": [["b","b",1571388944364000,"t",1571388944],
             ["v","v",1571388944364000,"t",1571388944]]}
  ]
  

  (It has range tombstones which are covered by the partition deletion)

Then, when we upgrade this node to 3.0 and try to read or run upgradesstables, we get this:

ERROR [node1_CompactionExecutor:1] node1 2019-10-18 10:44:11,325 DebuggableThreadPoolExecutor.java:242 - Error in ThreadPoolExecutor
java.lang.UnsupportedOperationException: null
	at org.apache.cassandra.db.LegacyLayout.extractStaticColumns(LegacyLayout.java:779) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:120) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:57) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:362) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:103) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:94) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:442) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:144) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:227) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:190) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:675) ~[dtest-3.0.19.jar:na]
	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[dtest-3.0.19.jar:na]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83) [dtest-3.0.19.jar:na]
	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]