Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-16495

Scheduled (Delayed) Schema Pull Tasks May Run After MIGRATION Stage Shutdown During Decommission

    XMLWordPrintableJSON

    Details

      Description

      A new test added in CASSANDRA-16181 stumbled across this, although it doesn’t happen consistently. When failure occurs, it appears to be because a delayed schema pull happens after decommission shuts down the MIGRATION stage’s thread pool.

      ERROR [node1_isolatedExecutor:1] node1 2021-02-15 19:35:36,284 CassandraDaemon.java:579 - Exception in thread Thread[node1_NonPeriodicTasks:1,5,node1] java.util.concurrent.RejectedExecutionException: ThreadPoolExecutor has shut down at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor$1.rejectedExecution(DebuggableThreadPoolExecutor.java:72) 
      at java.base/java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:825) 
      at java.base/java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1355) 
      at org.apache.cassandra.concurrent.DebuggableThreadPoolExecutor.execute(DebuggableThreadPoolExecutor.java:176) 
      at java.base/java.util.concurrent.AbstractExecutorService.submit(AbstractExecutorService.java:118) at org.apache.cassandra.concurrent.Stage.submit(Stage.java:129) 
      at org.apache.cassandra.schema.MigrationCoordinator.lambda$scheduleSchemaPull$2(MigrationCoordinator.java:362) 
      at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) 
      at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:304) 
      at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) 
      at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) 
      at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) 
      at java.base/java.lang.Thread.run(Thread.java:834)
      

      A fix might be as simple as shutting down ScheduledExecutors.nonPeriodicTasks in StorageService#decommission(). See the original discussion here.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                e.dimitrova Ekaterina Dimitrova
                Reporter:
                maedhroz Caleb Rackliffe
                Authors:
                Ekaterina Dimitrova
                Reviewers:
                Caleb Rackliffe, Yifan Cai
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: