[CASSANDRA-15800] BinLog deadlock on stopping when the sample queue is full

• Type: Bug
• Status: Resolved
• Priority: Normal
• Resolution: Not A Bug
• Component/s: Legacy/Tools, Observability/Logging, Tool/fql
• Bug Category: Degradation - Resource Management
• Severity: Normal
• Complexity: Normal
• Discovered By: Code Inspection
• Platform: All
• Assignee: Yifan Cai
• Reporter: Yifan Cai

Authors

Yifan Cai

• Created: 09/May/20 23:42
• Updated: 11/May/20 07:33
• Resolved: 11/May/20 07:33

Description

A deadlock can happen when 1) the BinLog is being stoped and 2) the BinLog's internal sample queue is full.

When stopping, BinLog first set the flag shouldContinue to false, so that the internal consumer thread stop consuming. It is possible to leave the queue being full.
Then, the BinLog puts one extra object NO_OP into the sample queue. However, the queue is already full, so the put operation blocks, and the stop method never returns.
Therefore, we got a deadlock.

BinLog is used by Cassandra 40 new features such as audit logging and full query logging.
If such deadlock happens, the thread cannot be not joined and the referenced items in the queue are never released, hence memory leak.