Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-11158

AssertionError: null in Slice$Bound.create

    XMLWordPrintableJSON

    Details

    • Severity:
      Critical

      Description

      We've been running Cassandra 3.0.2 for around a week now. Yesterday, we had a network event that briefly isolated one node from others in a 3 node cluster. Since then, we've been seeing a constant stream of "Finished hinted handoff" messages, as well as:

      WARN  16:34:39 Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {}
      java.lang.AssertionError: null
      at org.apache.cassandra.db.Slice$Bound.create(Slice.java:365) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.Slice$Bound$Serializer.deserializeValues(Slice.java:553) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.ClusteringPrefix$Serializer.deserialize(ClusteringPrefix.java:274) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.Serializers$2.deserialize(Serializers.java:115) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.Serializers$2.deserialize(Serializers.java:107) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo$Serializer.deserialize(IndexHelper.java:149) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:218) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.io.sstable.format.big.BigTableReader.getPosition(BigTableReader.java:216) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.io.sstable.format.SSTableReader.getPosition(SSTableReader.java:1568) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.columniterator.SSTableIterator.<init>(SSTableIterator.java:36) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.io.sstable.format.big.BigTableReader.iterator(BigTableReader.java:62) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndSSTablesInTimestampOrder(SinglePartitionReadCommand.java:715) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDiskInternal(SinglePartitionReadCommand.java:482) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.SinglePartitionReadCommand.queryMemtableAndDisk(SinglePartitionReadCommand.java:459) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.SinglePartitionReadCommand.queryStorage(SinglePartitionReadCommand.java:325) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.ReadCommand.executeLocally(ReadCommand.java:350) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.db.ReadCommandVerbHandler.doVerb(ReadCommandVerbHandler.java:45) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_72]
      at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-3.0.2.jar:3.0.2]
      at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-3.0.2.jar:3.0.2]
      at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
      

      and also

      ERROR 06:10:11 Exception in thread Thread[CompactionExecutor:1,1,main]
      java.lang.AssertionError: null
      at org.apache.cassandra.db.Slice$Bound.create(Slice.java:365) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.Slice$Bound$Serializer.deserializeValues(Slice.java:553) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.ClusteringPrefix$Serializer.deserialize(ClusteringPrefix.java:274) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.Serializers$2.deserialize(Serializers.java:115) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.Serializers$2.deserialize(Serializers.java:107) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.io.sstable.IndexHelper$IndexInfo$Serializer.deserialize(IndexHelper.java:149) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:218) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:305) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.io.sstable.format.big.BigTableScanner$KeyScanningIterator.computeNext(BigTableScanner.java:260) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.io.sstable.format.big.BigTableScanner.hasNext(BigTableScanner.java:240) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.utils.MergeIterator$Candidate.advance(MergeIterator.java:369) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.utils.MergeIterator$ManyToOne.advance(MergeIterator.java:189) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:158) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.utils.AbstractIterator.hasNext(AbstractIterator.java:47) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.partitions.UnfilteredPartitionIterators$2.hasNext(UnfilteredPartitionIterators.java:150) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.transform.BasePartitions.hasNext(BasePartitions.java:72) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.compaction.CompactionIterator.hasNext(CompactionIterator.java:226) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.compaction.CompactionTask.runMayThrow(CompactionTask.java:177) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:78) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionCandidate.run(CompactionManager.java:263) ~[apache-cassandra-3.0.3.jar:3.0.3]
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_72]
      at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[na:1.8.0_72]
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[na:1.8.0_72]
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [na:1.8.0_72]
      at java.lang.Thread.run(Thread.java:745) [na:1.8.0_72]
      

      on all 3 nodes. I'm now upgrading the nodes to 3.0.3, but the issue seems to persist.

        Attachments

          Activity

            People

            • Assignee:
              blambov Branimir Lambov
              Reporter:
              samukallio Samu Kallio
              Authors:
              Branimir Lambov
              Reviewers:
              Sylvain Lebresne
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: