[CASSANDRA-4981] Error when starting a node with vnodes while counter-add operations underway

• Type: Bug
• Status: Resolved
• Priority: Low
• Resolution: Cannot Reproduce
• Labels: qa-resolved

Environment

2-node cluster on ec2, ubuntu, cassandra-1.2.0 commit a32eb9f7d2f2868e8154d178e96e045859e1d855

• Severity: Low
• Assignee: Ryan McGuire
• Reporter: Tyler Patterson

Authors

Ryan McGuire

• Created: 21/Nov/12 17:25
• Updated: 16/Apr/19 09:32
• Resolved: 29/Oct/13 19:24

Description

Start both nodes, start stress on one node like this: "cassandra-stress --replication-factor=2 --operation=COUNTER_ADD"

While that is running: On the other node, kill cassandra, wait for "nodetool status" to show the node as down, and restart cassandra. I sometimes have to kill and restart cassandra several times to get the problem to happen.

I get this error several times in the log:

ERROR 15:39:33,198 Exception in thread Thread[MutationStage:16,5,main]
java.lang.AssertionError
	at org.apache.cassandra.locator.TokenMetadata.firstTokenIndex(TokenMetadata.java:748)
	at org.apache.cassandra.locator.TokenMetadata.firstToken(TokenMetadata.java:762)
	at org.apache.cassandra.locator.AbstractReplicationStrategy.getNaturalEndpoints(AbstractReplicationStrategy.java:95)
	at org.apache.cassandra.service.StorageService.getNaturalEndpoints(StorageService.java:2426)
	at org.apache.cassandra.service.StorageProxy.performWrite(StorageProxy.java:396)
	at org.apache.cassandra.service.StorageProxy.applyCounterMutationOnLeader(StorageProxy.java:755)
	at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:53)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:56)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:662)