[CASSANDRA-15295] Running into deadlock when do CommitLog initialization

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: 4.0-alpha3, 4.0
• Component/s: Local/Commit Log
• Bug Category: Availability - Unavailable
• Severity: Normal
• Complexity: Normal
• Discovered By: User Report
• Platform: All
• Since Version: 4.0

Source Control Link

https://github.com/apache/cassandra/commit/3a8300e0b86c4acfb7b7702197d36cc39ebe94bc

Test and Documentation Plan

Unspecified

• Assignee: Zephyr Guo Assign to me
• Reporter: Zephyr Guo

Authors

Dinesh Joshi, Zephyr Guo

Reviewers

Dinesh Joshi, Jordan West

• Created: 30/Aug/19 13:41
• Updated: 16/Mar/22 12:29
• Resolved: 12/Dec/19 18:17

Description

Recently, I found a cassandra(3.11.4) node stuck in STARTING status for a long time.
I used jstack to saw what happened. The main thread stuck in AbstractCommitLogSegmentManager.awaitAvailableSegment

The strange thing is COMMIT-LOG-ALLOCATOR thread state was runnable but it was not actually running. 

And then I used pstack to troubleshoot. I found COMMIT-LOG-ALLOCATOR block on java class initialization.

This is a deadlock obviously. CommitLog waits for a CommitLogSegment when initializing. In this moment, the CommitLog class is not initialized and the main thread holds the class lock. After that, COMMIT-LOG-ALLOCATOR creates a CommitLogSegment with exception and call CommitLog.handleCommitError(static method). COMMIT-LOG-ALLOCATOR will block on this line because CommitLog class is still initializing.

 

 

Issue Links

• causes: CASSANDRA-15692 NPE in ConnectionBurnTest
• links to: pull request