[CASSANDRA-6742] Make it safe to concurrently access ABSC after its construction

• Type: Bug
• Status: Resolved
• Priority: Urgent
• Resolution: Fixed
• Fix Version/s: 2.1 beta2
• Severity: Critical
• Assignee: Aleksey Yeschenko
• Reporter: Ryan McGuire

Authors

Aleksey Yeschenko

Reviewers

Benedict Elliott Smith

• Created: 19/Feb/14 22:13
• Updated: 16/Apr/19 09:31
• Resolved: 20/Feb/14 18:37

Description

This is a physical four node cluster. Configuration is attached.

Create a keyspace and table from the first node:

CREATE KEYSPACE "Keyspace1" WITH replication = {
  'class': 'SimpleStrategy',
  'replication_factor': '1'
};

USE "Keyspace1";

CREATE TABLE "Counter1" (
  key blob,
  column1 ascii,
  value counter,
  PRIMARY KEY (key, column1)
) WITH COMPACT STORAGE AND
  bloom_filter_fp_chance=0.010000 AND
  caching='KEYS_ONLY' AND
  comment='' AND
  dclocal_read_repair_chance=0.000000 AND
  gc_grace_seconds=864000 AND
  index_interval=128 AND
  read_repair_chance=0.100000 AND
  replicate_on_write='true' AND
  populate_io_cache_on_flush='false' AND
  default_time_to_live=0 AND
  speculative_retry='NONE' AND
  memtable_flush_period_in_ms=0 AND
  compaction={'class': 'SizeTieredCompactionStrategy'} AND
  compression={};

And the find the following in the logs:

INFO  [Thrift:1] 2014-02-19 14:04:35,828 MigrationManager.java:210 - Create new ColumnFamily: org.apache.cassandra.config.CFMetaData@d824292[cfId=d1bc0c30-99b1-11e3-a5f9-c187ff8103e2,ksName=Keyspace1,cfName=Counter1,cfType=Standard,comparator=org.apache.cassandra.db.marshal.AsciiType,comment=,readRepairChance=0.1,dclocalReadRepairChance=0.0,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.CounterColumnType,keyValidator=org.apache.cassandra.db.marshal.BytesType,minCompactionThreshold=4,maxCompactionThreshold=32,columnMetadata={java.nio.HeapByteBuffer[pos=0 lim=3 cap=3]=ColumnDefinition{name=key, type=org.apache.cassandra.db.marshal.BytesType, kind=PARTITION_KEY, componentIndex=null, indexName=null, indexType=null}, java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]=ColumnDefinition{name=value, type=org.apache.cassandra.db.marshal.CounterColumnType, kind=COMPACT_VALUE, componentIndex=null, indexName=null, indexType=null}, java.nio.HeapByteBuffer[pos=0 lim=7 cap=7]=ColumnDefinition{name=column1, type=org.apache.cassandra.db.marshal.AsciiType, kind=CLUSTERING_COLUMN, componentIndex=null, indexName=null, indexType=null}},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionParameters={},bloomFilterFpChance=0.01,memtableFlushPeriod=0,caching=KEYS_ONLY,defaultTimeToLive=0,minIndexInterval=128,maxIndexInterval=2048,speculativeRetry=NONE,populateIoCacheOnFlush=false,droppedColumns={},triggers={},rowsPerPartitionToCache=100]
ERROR [WRITE-/172.16.1.211] 2014-02-19 14:04:35,838 OutboundTcpConnection.java:256 - error writing to /172.16.1.211
java.lang.ArrayIndexOutOfBoundsException: -1
        at org.apache.cassandra.db.ArrayBackedSortedColumns.internalAppendOrReconcile(ArrayBackedSortedColumns.java:231) ~[main/:na]
        at org.apache.cassandra.db.ArrayBackedSortedColumns.sortCells(ArrayBackedSortedColumns.java:143) ~[main/:na]
        at org.apache.cassandra.db.ArrayBackedSortedColumns.maybeSortCells(ArrayBackedSortedColumns.java:103) ~[main/:na]
        at org.apache.cassandra.db.ArrayBackedSortedColumns.getColumnCount(ArrayBackedSortedColumns.java:313) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamilySerializer.contentSerializedSize(ColumnFamilySerializer.java:117) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamilySerializer.serializedSize(ColumnFamilySerializer.java:132) ~[main/:na]
        at org.apache.cassandra.db.Mutation$MutationSerializer.serializedSize(Mutation.java:337) ~[main/:na]
        at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.serializedSize(MigrationManager.java:397) ~[main/:na]
        at org.apache.cassandra.service.MigrationManager$MigrationsSerializer.serializedSize(MigrationManager.java:371) ~[main/:na]
        at org.apache.cassandra.net.MessageOut.serialize(MessageOut.java:116) ~[main/:na]
        at org.apache.cassandra.net.OutboundTcpConnection.writeInternal(OutboundTcpConnection.java:273) [main/:na]
        at org.apache.cassandra.net.OutboundTcpConnection.writeConnected(OutboundTcpConnection.java:225) [main/:na]
        at org.apache.cassandra.net.OutboundTcpConnection.run(OutboundTcpConnection.java:163) [main/:na]

Among a few other similar errors. See the attached log. There were no errors in the log of the node it was trying to contact.

Issue Links

• is duplicated by: CASSANDRA-6754 communication exceptions between nodes