[CASSANDRA-16495] Scheduled (Delayed) Schema Pull Tasks May Run After MIGRATION Stage Shutdown During Decommission

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 3.0.25, 3.11.11, 4.0-rc1, 4.0
• Component/s: Local/Startup and Shutdown
• Bug Category: Correctness - API / Semantic Implementation
• Severity: Low
• Complexity: Low Hanging Fruit
• Discovered By: Unit Test
• Platform: All

Source Control Link

https://github.com/apache/cassandra/commit/f2ff96eeaf0f481ebd1fd7314f4caef5a0f75ca9

Test and Documentation Plan

https://issues.apache.org/jira/browse/CASSANDRA-16495?focusedCommentId=17308927&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17308927

• Assignee: Ekaterina Dimitrova
• Reporter: Caleb Rackliffe

Authors

Ekaterina Dimitrova

Reviewers

Caleb Rackliffe, Yifan Cai

• Created: 05/Mar/21 21:48
• Updated: 25/Apr/21 11:32
• Resolved: 07/Apr/21 20:06

Description

A new test added in CASSANDRA-16181 stumbled across this, although it doesn’t happen consistently. When failure occurs, it appears to be because a delayed schema pull happens after decommission shuts down the MIGRATION stage’s thread pool.

ERROR [node1_isolatedExecutor:1] node1 2021-02-15 19:35:36,284 CassandraDaemon.java:579 - Exception in thread Thread[node1_NonPeriodicTasks:1,5,node1] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:72) 
at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) 
at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) 
at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:176) 
at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at org.apache.cassandra.concurrent.Stage.submit(Stage.java:129) 
at org.apache.cassandra.schema.MigrationCoordinator.lambda$scheduleSchemaPull$2(MigrationCoordinator.java:362) 
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) 
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 
at java.base/java.lang.Thread.run(Thread.java:834)

A fix might be as simple as shutting down ScheduledExecutors.nonPeriodicTasks in StorageService#decommission(). See the original discussion here.

Issue Links

• is related to: CASSANDRA-16568 Fix test shouldStreamHintsDuringDecommission - org.apache.cassandra.distributed.test.HintedHandoffAddRemoveNodesTest
• is related to: CASSANDRA-16181 4.0 Quality: Replication Test Audit