Cassandra
  1. Cassandra
  2. CASSANDRA-4289

Secondary Indexes fail following a system restart

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Fix Version/s: 1.2.0 beta 1
    • Component/s: Core
    • Labels:
      None
    • Environment:

      Single node, trunk

      Description

      Create a new cf with a secondary index, and queries with indexes predicates work fine until the server is restarted, after which they error and the following stacktrace is output to the log:

      java.lang.ClassCastException: java.math.BigInteger cannot be cast to java.nio.ByteBuffer
      	at org.apache.cassandra.dht.LocalToken.compareTo(LocalToken.java:44)
      	at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:88)
      	at org.apache.cassandra.db.DecoratedKey.compareTo(DecoratedKey.java:1)
      	at org.apache.cassandra.utils.IntervalTree.comparePoints(IntervalTree.java:191)
      	at org.apache.cassandra.utils.IntervalTree.contains(IntervalTree.java:203)
      	at org.apache.cassandra.utils.IntervalTree.access$3(IntervalTree.java:201)
      	at org.apache.cassandra.utils.IntervalTree$IntervalNode.searchInternal(IntervalTree.java:293)
      	at org.apache.cassandra.utils.IntervalTree.search(IntervalTree.java:140)
      	at org.apache.cassandra.utils.IntervalTree.search(IntervalTree.java:146)
      	at org.apache.cassandra.db.ColumnFamilyStore.markReferenced(ColumnFamilyStore.java:1259)
      	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:229)
      	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
      	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1300)
      	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1174)
      	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1104)
      	at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:144)
      	at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:1)
      	at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
      	at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
      	at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1409)
      	at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:88)
      	at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:595)
      	at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1398)
      	at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:47)
      	at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:870)
      	at org.apache.cassandra.cql3.statements.SelectStatement.multiRangeSlice(SelectStatement.java:259)
      	at org.apache.cassandra.cql3.statements.SelectStatement.execute(SelectStatement.java:134)
      	at org.apache.cassandra.cql3.QueryProcessor.processStatement(QueryProcessor.java:108)
      	at org.apache.cassandra.cql3.QueryProcessor.process(QueryProcessor.java:121)
      	at org.apache.cassandra.thrift.CassandraServer.execute_cql_query(CassandraServer.java:1236)
      	at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:3542)
      	at org.apache.cassandra.thrift.Cassandra$Processor$execute_cql_query.getResult(Cassandra.java:1)
      	at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
      	at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
      	at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:184)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
      	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
      	at java.lang.Thread.run(Thread.java:662)
      

      Tested with a single node setup & verified that this behaviour is only present in trunk, cassandra-1.0.10 works as expected.

        Issue Links

          Activity

          Hide
          Dave Brosius added a comment -

          easily reproducable. The index's column family's IntervalTree's head BigIntegerToken compared to the searched against column value.

          Show
          Dave Brosius added a comment - easily reproducable. The index's column family's IntervalTree's head BigIntegerToken compared to the searched against column value.
          Hide
          Dave Brosius added a comment -

          It appears this commit introduced this regression

          Save IndexSummary into new SSTable 'Summary' component for CASSANDRA-2392

          http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=04874186892c86a20181a2f64c5dc24285021b2c

          Show
          Dave Brosius added a comment - It appears this commit introduced this regression Save IndexSummary into new SSTable 'Summary' component for CASSANDRA-2392 http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a=commit;h=04874186892c86a20181a2f64c5dc24285021b2c
          Hide
          Pavel Yaskevich added a comment -

          The commit you mentioned doesn't touch CFS it merely saves/loads primary index into separate component, but anyway - what do you suggest as work around?

          Show
          Pavel Yaskevich added a comment - The commit you mentioned doesn't touch CFS it merely saves/loads primary index into separate component, but anyway - what do you suggest as work around?
          Hide
          Jonathan Ellis added a comment -

          What am I doing wrong?

          cqlsh> create schema ks with strategy_class ='SimpleStrategy' and strategy_options:replication_factor=1;
          cqlsh> use ks;
          cqlsh:ks> create table foo(key text primary key, i int);
          cqlsh:ks> create index i_idx on foo(i);
          cqlsh:ks> insert into foo (key, i) values('asdf', 1);
          cqlsh:ks> select * from foo where i = 1;
            KEY | i
          ------+---
           asdf | 1
          
          [restart Cassandra server and cqlsh]
          
          cqlsh> use ks;
          cqlsh:ks> select * from foo where i = 1;
           KEY  | i 
          ------+---
           asdf | 1
          
          Show
          Jonathan Ellis added a comment - What am I doing wrong? cqlsh> create schema ks with strategy_class ='SimpleStrategy' and strategy_options:replication_factor=1; cqlsh> use ks; cqlsh:ks> create table foo(key text primary key, i int); cqlsh:ks> create index i_idx on foo(i); cqlsh:ks> insert into foo (key, i) values('asdf', 1); cqlsh:ks> select * from foo where i = 1; KEY | i ------+--- asdf | 1 [restart Cassandra server and cqlsh] cqlsh> use ks; cqlsh:ks> select * from foo where i = 1; KEY | i ------+--- asdf | 1
          Hide
          Dave Brosius added a comment -

          before stopping the server do

          nodetool flush

          Show
          Dave Brosius added a comment - before stopping the server do nodetool flush
          Hide
          Jonathan Ellis added a comment -

          Still works for me.

          I did fix a bug in the key cache for index CFs (attached) but I don't think that's what you're seeing.

          Show
          Jonathan Ellis added a comment - Still works for me. I did fix a bug in the key cache for index CFs (attached) but I don't think that's what you're seeing.
          Hide
          Sam Tunnicliffe added a comment -

          SSTableReader.loadSummary wasn't taking the specific IPartitioner for the sstable into consideration, so when the IndexSummary for a secondary index cf was loaded from disk, the keys in the summary and the first & last keys of the sstable were decorated incorrectly.

          jbellis: I guess you weren't seeing this as the IndexSummary wasn't being written down to disk, but I don't why that would be

          Show
          Sam Tunnicliffe added a comment - SSTableReader.loadSummary wasn't taking the specific IPartitioner for the sstable into consideration, so when the IndexSummary for a secondary index cf was loaded from disk, the keys in the summary and the first & last keys of the sstable were decorated incorrectly. jbellis: I guess you weren't seeing this as the IndexSummary wasn't being written down to disk, but I don't why that would be
          Hide
          Pavel Yaskevich added a comment -

          Committed combination of Sam's patch and changes to CacheService so row/key cache serializer methods would use per-CF partitioner instead of global.

          Show
          Pavel Yaskevich added a comment - Committed combination of Sam's patch and changes to CacheService so row/key cache serializer methods would use per-CF partitioner instead of global.

            People

            • Assignee:
              Sam Tunnicliffe
              Reporter:
              Sam Tunnicliffe
              Reviewer:
              Pavel Yaskevich
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development