[CASSANDRA-10579] IndexOutOfBoundsException during memtable flushing at startup (with offheap_objects)

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 2.1.12, 2.2.4

Environment

2.1.10 on linux

• Severity: Normal
• Assignee: Benedict Elliott Smith
• Reporter: Jeff Griffith

Authors

Benedict Elliott Smith

Reviewers

Branimir Lambov

• Created: 23/Oct/15 15:20
• Updated: 16/Apr/19 09:30
• Resolved: 02/Nov/15 19:47

Description

Sometimes we have problems at startup where memtable flushes with an index out of bounds exception as seen below. Cassandra is then dead in the water until we track down the corresponding commit log via the segment ID and remove it:

INFO  [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log
INFO  [main] 2015-10-23 14:43:36,440 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log (CL version 4, messaging version 8)
INFO  [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:478 - Finished reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832692.log
INFO  [main] 2015-10-23 14:43:36,594 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log
INFO  [main] 2015-10-23 14:43:36,595 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log (CL version 4, messaging version 8)
INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:478 - Finished reading /home/y/var/cassandra/commitlog/CommitLog-4-1445474832693.log
INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:267 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log
INFO  [main] 2015-10-23 14:43:36,699 CommitLogReplayer.java:270 - Replaying /home/y/var/cassandra/commitlog/CommitLog-4-1445474832694.log (CL version 4, messaging version 8)
WARN  [SharedPool-Worker-5] 2015-10-23 14:43:36,747 AbstractTracingAwareExecutorService.java:169 - Uncaught exception on thread Thread[SharedPool-Worker-5,5,main]: {}
java.lang.ArrayIndexOutOfBoundsException: 6
        at org.apache.cassandra.db.AbstractNativeCell.nametype(AbstractNativeCell.java:204) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.AbstractNativeCell.isStatic(AbstractNativeCell.java:199) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.composites.AbstractCType.compare(AbstractCType.java:166) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:61) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.composites.AbstractCellNameType$1.compare(AbstractCellNameType.java:58) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.btree.BTree.find(BTree.java:277) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.btree.NodeBuilder.update(NodeBuilder.java:154) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.btree.Builder.update(Builder.java:74) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.btree.BTree.update(BTree.java:186) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.AtomicBTreeColumns.addAllWithSizeDelta(AtomicBTreeColumns.java:225) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.Memtable.put(Memtable.java:210) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:1225) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:396) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:359) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:455) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[na:1.8.0_31]
        at org.apache.cassandra.concurrent.AbstractTracingAwareExecutorService$FutureTask.run(AbstractTracingAwareExecutorService.java:164) ~[apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:105) [apache-cassandra-2.1.10.jar:2.1.10-SNAPSHOT]
        at java.lang.Thread.run(Thread.java:745) [na:1.8.0_31]

Issue Links

• is duplicated by: CASSANDRA-10626 Seeing repeated Uncaught exception on thread Thread[SharedPool-Worker-1,5,main]: {} java.lang.AssertionError: null in logs