Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15363

Read repair in mixed mode between 2.1 and 3.0 on COMPACT STORAGE tables causes unreadable sstables after upgrade

    XMLWordPrintableJSON

    Details

      Description

      if we have a table like this:

      CREATE TABLE tbl (pk ascii, b boolean, v blob, PRIMARY KEY (pk)) WITH COMPACT STORAGE

      with a cluster where node1 is 2.1 and node2 is 3.0 (during upgrade):

      • node2 coordinates a delete DELETE FROM tbl WHERE pk = 'something' which node1 does not get
      • node1 coordinates a quorum read SELECT * FROM tbl WHERE id = 'something' which causes a read repair
      • this makes node1 flush an sstable like this:
        [
        {"key": "something",
         "metadata": {"deletionInfo": {"markedForDeleteAt":1571388944364000,"localDeletionTime":1571388944}},
         "cells": [["b","b",1571388944364000,"t",1571388944],
                   ["v","v",1571388944364000,"t",1571388944]]}
        ]
        

        (It has range tombstones which are covered by the partition deletion)

      Then, when we upgrade this node to 3.0 and try to read or run upgradesstables, we get this:

      ERROR [node1_CompactionExecutor:1] node1 2019-10-18 10:44:11,325 DebuggableThreadPoolExecutor.java:242 - Error in ThreadPoolExecutor
      java.lang.UnsupportedOperationException: null
      	at org.apache.cassandra.db.LegacyLayout.extractStaticColumns(LegacyLayout.java:779) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.io.sstable.SSTableSimpleIterator$OldFormatIterator.readStaticRow(SSTableSimpleIterator.java:120) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.io.sstable.SSTableIdentityIterator.<init>(SSTableIdentityIterator.java:57) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator$1.initializeIterator(BigTableScanner.java:362) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.maybeInit(LazilyInitializedUnfilteredRowIterator.java:48) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.isReverseOrder(LazilyInitializedUnfilteredRowIterator.java:65) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:103) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$1.reduce(UnfilteredPartitionIterators.java:94) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.utils.MergeIterator$OneToOne.computeNext(MergeIterator.java:442) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:144) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:92) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:227) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:190) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:675) ~[dtest-3.0.19.jar:na]
      	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[dtest-3.0.19.jar:na]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_121]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_121]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_121]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_121]
      	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83) [dtest-3.0.19.jar:na]
      	at java.lang.Thread.run(Thread.java:745) ~[na:1.8.0_121]
      

        Attachments

          Activity

            People

            • Assignee:
              marcuse Marcus Eriksson
              Reporter:
              marcuse Marcus Eriksson
              Authors:
              Marcus Eriksson
              Reviewers:
              Aleksey Yeschenko, Benedict Elliott Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: