[CASSANDRA-6589] Gossip AssertionError: Added cell does not sort as the last cell

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Cannot Reproduce

Environment

Debian Wheezy amd64
Oracle JDK 1.7_45
Branch: trunk

• Severity: Normal
• Since Version: 2.1 rc3
• Assignee: Michael Shuler
• Reporter: Michael Shuler

Authors

Michael Shuler

• Created: 15/Jan/14 16:54
• Updated: 16/Apr/19 09:31
• Resolved: 18/Dec/14 22:59

Description

The super_counter_test.py dtest sometimes, but not always, hangs until killed (^C) when running with vnodes. Reproduction (again, sometimes this is successful - keep trying until it hangs for a few minutes, which is about 1/3-1/2 of the time for me):

$ ENABLE_VNODES=true nosetests -v super_counter_test.py

If it does hang up on you, in another shell:

$ tail -F /tmp/dtest-*/test/node*/logs/system.log

Scrolling errors similar to:

ERROR [GossipStage:54] 2014-01-15 10:39:30,229 CassandraDaemon.java:139 - Exception in thread Thread[GossipStage:54,5,main]
java.lang.AssertionError: Added cell does not sort as the last cell
        at org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:116) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamily.addColumn(ColumnFamily.java:118) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamily.addIfRelevant(ColumnFamily.java:112) ~[main/:na]
        at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:214) ~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:123) ~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:81) ~[main/:na]
        at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:73) ~[main/:na]
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:296) ~[main/:na]
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:52) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1522) ~[main/:na]
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1341) ~[main/:na]
        at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327) ~[main/:na]
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:59) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.readLocally(SelectStatement.java:223) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:241) ~[main/:na]
        at org.apache.cassandra.cql3.statements.SelectStatement.executeInternal(SelectStatement.java:57) ~[main/:na]
        at org.apache.cassandra.cql3.QueryProcessor.processInternal(QueryProcessor.java:261) ~[main/:na]
        at org.apache.cassandra.db.SystemKeyspace.getPreferredIP(SystemKeyspace.java:457) ~[main/:na]
        at org.apache.cassandra.net.OutboundTcpConnectionPool.<init>(OutboundTcpConnectionPool.java:48) ~[main/:na]
        at org.apache.cassandra.net.MessagingService.getConnectionPool(MessagingService.java:491) ~[main/:na]
        at org.apache.cassandra.net.MessagingService.getConnection(MessagingService.java:505) ~[main/:na]
        at org.apache.cassandra.net.MessagingService.sendOneWay(MessagingService.java:637) ~[main/:na]
        at org.apache.cassandra.net.MessagingService.sendReply(MessagingService.java:611) ~[main/:na]
        at org.apache.cassandra.service.EchoVerbHandler.doVerb(EchoVerbHandler.java:40) ~[main/:na]
        at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:60) ~[main/:na]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) ~[na:1.7.0_45]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) ~[na:1.7.0_45]
        at java.lang.Thread.run(Thread.java:744) ~[na:1.7.0_45]

along with node1 repeatedly marking node3 up and down.