[CASSANDRA-15592] IllegalStateException in gossip after removing node

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.21, 3.11.7, 4.0-alpha4, 4.0
• Component/s: Cluster/Gossip
• Bug Category: Correctness - API / Semantic Implementation
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 3.0.19

Source Control Link

https://github.com/apache/cassandra/commit/c2bca69734253bea01bab59285df2b3ab0e571cb

Test and Documentation Plan

test added

• Assignee: Marcus Olsson
• Reporter: Marcus Olsson

Authors

Marcus Olsson

Reviewers

Brandon Williams

• Created: 20/Feb/20 12:05
• Updated: 07/Jul/20 15:16
• Resolved: 21/Feb/20 15:23

Description

In one of our test environments we encountered the following exception:

2020-02-02T10:50:13.276+0100 [GossipTasks:1] ERROR o.a.c.u.NoSpamLogger$NoSpamLogStatement:97 log java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1
 at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178)
 at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465)
 at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895)
 at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78)
 at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240)
 at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84)
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
 at java.lang.Thread.run(Thread.java:748)
java.lang.IllegalStateException: Attempting gossip state mutation from illegal thread: GossipTasks:1
 at org.apache.cassandra.gms.Gossiper.checkProperThreadForStateMutation(Gossiper.java:178) [apache-cassandra-3.11.5.jar:3.11.5]
 at org.apache.cassandra.gms.Gossiper.evictFromMembership(Gossiper.java:465) [apache-cassandra-3.11.5.jar:3.11.5]
 at org.apache.cassandra.gms.Gossiper.doStatusCheck(Gossiper.java:895) [apache-cassandra-3.11.5.jar:3.11.5]
 at org.apache.cassandra.gms.Gossiper.access$700(Gossiper.java:78) [apache-cassandra-3.11.5.jar:3.11.5]
 at org.apache.cassandra.gms.Gossiper$GossipTask.run(Gossiper.java:240) [apache-cassandra-3.11.5.jar:3.11.5]
 at org.apache.cassandra.concurrent.DebuggableScheduledThreadPoolExecutor$UncomplainingRunnable.run(DebuggableScheduledThreadPoolExecutor.java:118) [apache-cassandra-3.11.5.jar:3.11.5]
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [na:1.8.0_231]
 at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308) [na:1.8.0_231]
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180) [na:1.8.0_231]
 at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294) [na:1.8.0_231]
 at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [na:1.8.0_231]
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [na:1.8.0_231]
 at org.apache.cassandra.concurrent.NamedThreadFactory.lambda$threadLocalDeallocator$0(NamedThreadFactory.java:84) [apache-cassandra-3.11.5.jar:3.11.5]
 at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) ~[netty-all-4.1.42.Final.jar:4.1.42.Final]
 at java.lang.Thread.run(Thread.java:748) ~[na:1.8.0_231]

Since CASSANDRA-15059 we check that all state changes are performed in the GossipStage but it seems like it was still performed in the "current" thread here. It should be as simple as adding a

runInGossipStageBlocking(() ->)

for it.

I'll upload patches for 3.0, 3.11 and 4.0.