Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-15799

CorruptSSTableException when compacting a 3.0 format sstable that was originally created as an outcome of 2.1 sstable upgrade

    XMLWordPrintableJSON

    Details

    • Bug Category:
      Correctness - Unrecoverable Corruption / Loss
    • Severity:
      Critical
    • Complexity:
      Normal
    • Discovered By:
      User Report
    • Platform:
      All
    • Impacts:
      None

      Description

      Below is the exception with stack trace. This issue is reproduce-able.

      DEBUG [CompactionExecutor:10] 2020-05-07 19:33:34,268 CompactionTask.java:158 - Compacting (a3ea9fc0-9099-11ea-933f-c5e852f71338) [/mnt/data/cassandra/data/ks/cf/md-10802-big-Data.db:level=0, ]
      ERROR [CompactionExecutor:10] 2020-05-07 19:33:34,275 CassandraDaemon.java:208 - Exception in thread Thread[CompactionExecutor:10,1,RMI Runtime]
      org.apache.cassandra.io.sstable.CorruptSSTableException: Corrupted: /mnt/data/cassandra/data/ks/cf/md-10802-big-Data.db
      	at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:105) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:30) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:95) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.rows.LazilyInitializedUnfilteredRowIterator.computeNext(LazilyInitializedUnfilteredRowIterator.java:32) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.utils.MergeIterator$TrivialOneToOne.computeNext(MergeIterator.java:460) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:534) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.rows.UnfilteredRowIterators$UnfilteredRowMergeIterator.computeNext(UnfilteredRowIterators.java:394) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.transform.BaseRows.hasNext(BaseRows.java:129) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.ColumnIndex$Builder.build(ColumnIndex.java:111) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.ColumnIndex.writeAndBuildIndex(ColumnIndex.java:52) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.io.sstable.format.big.BigTableWriter.append(BigTableWriter.java:165) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.io.sstable.SSTableRewriter.append(SSTableRewriter.java:126) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.compaction.writers.DefaultCompactionWriter.realAppend(DefaultCompactionWriter.java:57) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.compaction.writers.CompactionAwareWriter.append(CompactionAwareWriter.java:109) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:195) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:89) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:61) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.compaction.CompactionManager$8.runMayThrow(CompactionManager.java:675) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_231]
      	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_231]
      	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[na:1.8.0_231]
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_231]
      	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:83) [nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_231]
      Caused by: java.lang.ArrayIndexOutOfBoundsException: 119
      	at org.apache.cassandra.db.RangeTombstone$Bound$Serializer.deserialize(RangeTombstone.java:201) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.rows.UnfilteredSerializer.deserializeOne(UnfilteredSerializer.java:407) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.db.rows.UnfilteredSerializer.deserialize(UnfilteredSerializer.java:373) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:87) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.io.sstable.SSTableSimpleIterator$CurrentFormatIterator.computeNext(SSTableSimpleIterator.java:65) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.io.sstable.SSTableIdentityIterator.doCompute(SSTableIdentityIterator.java:123) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	at org.apache.cassandra.io.sstable.SSTableIdentityIterator.computeNext(SSTableIdentityIterator.java:100) ~[nf-cassandra-3.0.19.8.jar:3.0.19.8]
      	... 29 common frames omitted
      

      Below are the logs that indicate the origin of this sstable

      DEBUG [CompactionExecutor:7] 2020-05-07 18:39:59,235 CompactionTask.java:158 - Compacting (279b8530-9092-11ea-933f-c5e852f71338) [/mnt/data/cassandra/data/ks/cf/ks-cf-ka-10799-Data.db:level=0, /mnt/data/cassandra/data/ks/cf/ks-cf-ka-10800-Data.db:level=0, /mnt/data/cassandra/data/ks/cf/ks-cf-ka-10801-Data.db:level=0, ]
      DEBUG [CompactionExecutor:7] 2020-05-07 18:40:01,053 CompactionTask.java:235 - Compacted (279b8530-9092-11ea-933f-c5e852f71338) 3 sstables to [/mnt/data/cassandra/data/ks/cf/md-10802-big,] to level=0.  35,435,268 bytes to 35,317,747 (~99% of original) in 1,817ms = 18.536945MB/s.  789 total partitions merged to 772.  Partition merge counts were {1:786, 3:1, }
      

        Attachments

        1. fake-deletedcell-if-bad.patch
          4 kB
          David Capwell

          Issue Links

            Activity

              People

              • Assignee:
                dcapwell David Capwell
                Reporter:
                sumanth.pasupuleti Sumanth Pasupuleti
                Authors:
                David Capwell
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated: