[CASSANDRA-16276] Drain and/or shutdown might throw because of slow messaging service shutdown

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-beta4, 4.0
• Component/s: Local/Startup and Shutdown
• Bug Category: Code - Bug - Unclear Impact
• Severity: Low
• Complexity: Low Hanging Fruit
• Discovered By: User Report
• Platform: All
• Since Version: 4.0-alpha1

Source Control Link

7ca997ba3514e19864d53b8ca56a1e4f5c26208f

Test and Documentation Plan

None needed

• Assignee: Aleksey Yeschenko
• Reporter: Aleksey Yeschenko

Authors

Aleksey Yeschenko

Reviewers

Aleksey Yeschenko, Alex Petrov, Marcus Eriksson

• Created: 16/Nov/20 16:05
• Updated: 03/Jan/21 16:58
• Resolved: 18/Nov/20 14:06

Description

If we invoke nodetool drain before shutdown, it sometimes fails to shut down messaging service in time (in this case - timing out the shutdown of the eventloopgroup by Netty). But, not before we manage to set isShutdown of StorageService to true, despite aborting further drain logic (including shutting down mutation stages).

Then, via on shutdown hook, we invoke drain() method again, implicitly. We see that the mutation stage is not shutdown and proceed to assert that isShutdown == false, failing that assertion and triggering a second error log message.

The patch merely ensures that any exception thrown by MS shutdown is captured so that drain logic can complete the first time around.