[CASSANDRA-15975] SEP.shutdownAndWait does not wait for termination and SEPExecutor.awaitTermination may hang

• Type: Bug
• Status: Review In Progress
• Priority: Normal
• Resolution: Unresolved
• Component/s: Local/Other
• Bug Category: Code
• Severity: Low
• Complexity: Normal
• Discovered By: Unit Test
• Platform: All

Test and Documentation Plan

PR / Branch
CircleCI - unit tests had an unrelated failure in org.apache.cassandra.streaming.async.StreamingInboundHandlerTest

• Assignee: Jon Meredith
• Reporter: Jon Meredith

Authors

Jon Meredith

Reviewers

Benedict Elliott Smith, David Capwell

• Created: 24/Jul/20 00:45
• Updated: 09/Nov/21 13:21

Description

SharedExecutorPool.shutdownAndWait calls shutdownNow on the executors it owns, which has the side effect of removing the executor from the list, then uses the same emptied list to try and wait for them to complete shutting down, which completes immediately.
 
Once fixed by copying the list in SEP.shutdownAndWait, the SEPExecutorTest suite fails with a timeout due to a startup race in SEPWorker.
 
If a SEPExecutor is shutdown immediately after maybeSchedule creates a new SEPWorker,
but before it is able to execute the first task then it exits immediately on entering the work loop with a work and task permit reserved for it, which invalidates the work/task permit accounting and prevents calling the shutdown SimpleCondition.signalAll when the last SEPWorker exits.