[CASSANDRA-12926] RejectedExecutionException logged as SEVERE when task already completed

• Type: Improvement
• Status: Resolved
• Priority: Low
• Resolution: Not A Problem
• Component/s: Legacy/Streaming and Messaging
• Labels: pull-request-available

Environment

2.1.16

• Assignee: Unassigned
• Reporter: Sebastien Bonnet
• Created: 17/Nov/16 16:06
• Updated: 16/Mar/22 11:47
• Resolved: 30/Jun/21 16:42

Description

The issue is triggered by the MessagingService expiring a repair Snapshot which Executor has already been shutdown and is an a TERMINATED state.

In this case trying to complete the SnapshotTask has little value, since the Task is already completed. Yet the error is reported as SEVERE by the guava library and shows in the journal logs.

Perhaps the Callback could check if the task is already complete before setting the exception?
https://github.com/apache/cassandra/blob/cassandra-2.1.16/src/java/org/apache/cassandra/repair/SnapshotTask.java#L78

Nov 15, 2016 8:49:31 PM com.google.common.util.concurrent.ExecutionList executeListener
SEVERE: RuntimeException while executing runnable com.google.common.util.concurrent.Futures$4@3df75dd9 with executor com.google.common.util.concurrent.MoreExecutors$ListeningDecorator@370f36a9
java.util.concurrent.RejectedExecutionException: Task com.google.common.util.concurrent.Futures$4@3df75dd9 rejected from java.util.concurrent.ThreadPoolExecutor@3330a417[Terminated, pool size = 0, active threads = 0, queued tasks = 0, completed tasks = 17]
        at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2047)
        at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:823)
        at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1369)
        at com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:484)
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
        at com.google.common.util.concurrent.Futures$CombinedFuture.setExceptionAndMaybeLog(Futures.java:1566)
        at com.google.common.util.concurrent.Futures$CombinedFuture.setOneValue(Futures.java:1617)
        at com.google.common.util.concurrent.Futures$CombinedFuture.access$400(Futures.java:1470)
        at com.google.common.util.concurrent.Futures$CombinedFuture$2.run(Futures.java:1548)
        at com.google.common.util.concurrent.MoreExecutors$SameThreadExecutorService.execute(MoreExecutors.java:297)
        at com.google.common.util.concurrent.ExecutionList.executeListener(ExecutionList.java:156)
        at com.google.common.util.concurrent.ExecutionList.execute(ExecutionList.java:145)
        at com.google.common.util.concurrent.AbstractFuture.setException(AbstractFuture.java:202)
        at org.apache.cassandra.repair.SnapshotTask.access$200(SnapshotTask.java:34)
        at org.apache.cassandra.repair.SnapshotTask$SnapshotCallback.onFailure(SnapshotTask.java:78)
        at org.apache.cassandra.net.MessagingService$5$1.run(MessagingService.java:374)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:745)

Issue Links

• links to: GitHub Pull Request #245