Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6733

Upgrade of 1.2.11 to 2.0.5 make IllegalArgumentException in Buffer.limit on read of a super column family

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Normal
    • Resolution: Fixed
    • 2.0.6
    • None
    • None
    • Normal

    Description

      We have a super column family which was first created with a 1.0.x. Then upgraded to 1.1.x, then to 1.2.11, and now to 2.0.5.

      cqlsh:QaUser> desc table user_view;
      CREATE TABLE user_view (
        key bigint,
        column1 varint,
        column2 text,
        value counter,
        PRIMARY KEY (key, column1, column2)
      ) WITH COMPACT STORAGE AND
        bloom_filter_fp_chance=0.010000 AND
        caching='KEYS_ONLY' AND
        comment='' AND
        dclocal_read_repair_chance=0.000000 AND
        gc_grace_seconds=864000 AND
        index_interval=128 AND
        read_repair_chance=1.000000 AND
        replicate_on_write='true' AND
        populate_io_cache_on_flush='false' AND
        default_time_to_live=0 AND
        speculative_retry='99.0PERCENTILE' AND
        memtable_flush_period_in_ms=0 AND
        compaction={'class': 'SizeTieredCompactionStrategy'} AND
        compression={'sstable_compression': 'SnappyCompressor'};
      

      With cqlsh, the following query was doing a timeout:

      select * from user_view where key = 3 and column1 = 1 and column2 = '20130218';
      

      In the log of cassandra, we could read:

      ERROR [ReadStage:1385] 2014-02-19 14:45:19,549 CassandraDaemon.java (line 192) Exception in thread Thread[ReadStage:1385,5,main]
      java.lang.IllegalArgumentException
              at java.nio.Buffer.limit(Buffer.java:267)
              at org.apache.cassandra.db.marshal.AbstractCompositeType.getBytes(AbstractCompositeType.java:55)
              at org.apache.cassandra.db.marshal.AbstractCompositeType.getWithShortLength(AbstractCompositeType.java:64)
              at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:82)
              at org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35)
              at org.apache.cassandra.db.marshal.AbstractType$1.compare(AbstractType.java:63)
              at org.apache.cassandra.db.marshal.AbstractType$1.compare(AbstractType.java:60)
              at java.util.Collections.indexedBinarySearch(Collections.java:377)
              at java.util.Collections.binarySearch(Collections.java:365)
              at org.apache.cassandra.io.sstable.IndexHelper.indexFor(IndexHelper.java:144)
              at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.setNextSlice(IndexedSliceReader.java:262)
              at org.apache.cassandra.db.columniterator.IndexedSliceReader$IndexedBlockFetcher.<init>(IndexedSliceReader.java:255)
              at org.apache.cassandra.db.columniterator.IndexedSliceReader.<init>(IndexedSliceReader.java:91)
              at org.apache.cassandra.db.columniterator.SSTableSliceIterator.createReader(SSTableSliceIterator.java:65)
              at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:42)
              at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:167)
              at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:62)
              at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:250)
              at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
              at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1560)
              at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
              at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
              at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
              at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:47)
              at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:724)
      

      I tried launching repair on our 2 nodes, nothing improved.
      I tried launching a major compaction on this column family, the query doesn't fail anymore and return expected results;

      This happens on our cluster which is used for integration and test purpose, not much activity on it. There are only 2 nodes and the replication factor is at 1. Since it is our test cluster, I have a quite small (2 x ~500K) snapshot done before the upgrade of the cluster I could share, if needed.

      Attachments

        1. QaUser_user_view_node2.tgz
          77 kB
          Nicolas Lalevée
        2. QaUser_user_view_node1.tgz
          161 kB
          Nicolas Lalevée
        3. 6733.txt
          6 kB
          Sylvain Lebresne

        Activity

          People

            slebresne Sylvain Lebresne
            hibou Nicolas Lalevée
            Sylvain Lebresne
            Vijay
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: