[CASSANDRA-6733] Upgrade of 1.2.11 to 2.0.5 make IllegalArgumentException in Buffer.limit on read of a super column family

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.0.6
• Severity: Normal
• Assignee: Sylvain Lebresne
• Reporter: Nicolas Lalevée

Authors

Sylvain Lebresne

Reviewers

Vijay

• Created: 19/Feb/14 14:18
• Updated: 16/Apr/19 09:31
• Resolved: 04/Mar/14 10:56

Description

We have a super column family which was first created with a 1.0.x. Then upgraded to 1.1.x, then to 1.2.11, and now to 2.0.5.

cqlsh:QaUser> desc table user_view;
CREATE TABLE user_view (
  key bigint,
  column1 varint,
  column2 text,
  value counter,
  PRIMARY KEY (key, column1, column2)
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=1.000000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='99.0PERCENTILE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={'sstable_compression': 'SnappyCompressor'};

With cqlsh, the following query was doing a timeout:

select * from user_view where key = 3 and column1 = 1 and column2 = '20130218';

In the log of cassandra, we could read:

ERROR [ReadStage:1385] 2014-02-19 14:45:19,549 CassandraDaemon.java (line 192) Exception in thread Thread[ReadStage:1385,5,main]
java.lang.IllegalArgumentException
        at java.nio.Buffer.limit(Buffer.java:267)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:55)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:64)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:82)
        at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35)
        at org.apache.cassandra.db.marshal.AbstractType$1.compare(AbstractType.java:63)
        at org.apache.cassandra.db.marshal.AbstractType$1.compare(AbstractType.java:60)
        at java.util.Collections.indexedBinarySearch(Collections.java:377)
        at java.util.Collections.binarySearch(Collections.java:365)
        at org.apache.cassandra.io.sstable.IndexHelper.indexFor(IndexHelper.java:144)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.setNextSlice(IndexedSliceReader.java:262)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.<init>(IndexedSliceReader.java:255)
        at org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:91)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42)
        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1560)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
        at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47)
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:724)

I tried launching repair on our 2 nodes, nothing improved.
I tried launching a major compaction on this column family, the query doesn't fail anymore and return expected results;

This happens on our cluster which is used for integration and test purpose, not much activity on it. There are only 2 nodes and the replication factor is at 1. Since it is our test cluster, I have a quite small (2 x ~500K) snapshot done before the upgrade of the cluster I could share, if needed.