[CASSANDRA-13660] Correctly timed kill -9 can put incremental repair sessions in an illegal state

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-alpha1, 4.0
• Severity: Normal
• Assignee: Blake Eggleston
• Reporter: Blake Eggleston

Authors

Blake Eggleston

Reviewers

Marcus Eriksson

• Created: 03/Jul/17 22:07
• Updated: 15/May/20 07:59
• Resolved: 06/Jul/17 17:20

Description

If a node is killed after it has sent a finalize promise message to an incremental repair coordinator, but before that section of commit log has been synced to disk, it can startup with the incremental repair session in a previous state, leading the following exception:

java.lang.RuntimeException: java.lang.IllegalArgumentException: Invalid state transition PREPARED -> FINALIZED
	at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:201) ~[cie-cassandra-3.0.13.5.jar:3.0.13.5]
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:67) ~[cie-cassandra-3.0.13.5.jar:3.0.13.5]
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) ~[?:1.8.0_112]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) ~[?:1.8.0_112]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) ~[?:1.8.0_112]
	at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:79) ~[cie-cassandra-3.0.13.5.jar:3.0.13.5]
	at java.lang.Thread.run(Thread.java:745) [?:1.8.0_112]
Caused by: java.lang.IllegalArgumentException: Invalid state transition PREPARED -> FINALIZED
	at com.google.common.base.Preconditions.checkArgument(Preconditions.java:145) ~[guava-18.0.jar:?]
	at org.apache.cassandra.repair.consistent.LocalSessions.setStateAndSave(LocalSessions.java:452) ~[cie-cassandra-3.0.13.5.jar:3.0.13.5]
	at org.apache.cassandra.repair.consistent.LocalSessions.handleStatusResponse(LocalSessions.java:679) ~[cie-cassandra-3.0.13.5.jar:3.0.13.5]
	at org.apache.cassandra.repair.RepairMessageVerbHandler.doVerb(RepairMessageVerbHandler.java:188) ~[cie-cassandra-3.0.13.5.jar:3.0.13.5]
	... 7 more