[HDDS-5122] SCM Reinitialization can end up leaking Ratis Segmented RaftLogWorker threads

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.2.0
• Component/s: SCM HA
• Labels: pull-request-available
• Assignee: Shashikant Banerjee
• Reporter: István Fajth
• Created: 20/Apr/21 07:50
• Updated: 22/Apr/21 08:02
• Resolved: 22/Apr/21 08:02

Description

During SCM reinitialialisation, ratis server is spinned up to check if an existing ratis group exists or not, and closes the server without starting it. In ratis, the segmented raft log worker thraeds are started during init() itself but get closed during raftServer.close() only if the server transitions to RUNNING state which causes the issue.

 

Attaching to process ID 266710, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.232-b09
Deadlock Detection:No deadlocks found.Thread 266745: (state = BLOCKED)Locked ownable synchronizers:
    - NoneThread 266783: (state = BLOCKED)
 - sun.misc.Unsafe.park(boolean, long) @bci=0 (Compiled frame; information may be imprecise)
 - java.util.concurrent.locks.LockSupport.parkNanos(java.lang.Object, long) @bci=20, line=215 (Compiled frame)
 - java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(long) @bci=78, line=2078 (Compiled frame)
 - org.apache.ratis.util.DataBlockingQueue.poll(org.apache.ratis.util.TimeDuration) @bci=134, line=137 (Compiled frame)
 - org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker.run() @bci=16, line=292 (Interpreted frame)
 - org.apache.ratis.server.raftlog.segmented.SegmentedRaftLogWorker$$Lambda$161.run() @bci=4 (Interpreted frame)
 - java.lang.Thread.run() @bci=11, line=748 (Interpreted frame)Locked ownable synchronizers:
    - NoneThread 266761: (state = BLOCKED)Locked ownable synchronizers:
    - NoneThread 266760: (state = BLOCKED)Locked ownable synchronizers:
    - NoneThread 266759: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.ref.ReferenceQueue.remove(long) @bci=59, line=144 (Compiled frame)
 - java.lang.ref.ReferenceQueue.remove() @bci=2, line=165 (Compiled frame)
 - java.lang.ref.Finalizer$FinalizerThread.run() @bci=36, line=216 (Interpreted frame)Locked ownable synchronizers:
    - NoneThread 266758: (state = BLOCKED)
 - java.lang.Object.wait(long) @bci=0 (Interpreted frame)
 - java.lang.Object.wait() @bci=2, line=502 (Compiled frame)
 - java.lang.ref.Reference.tryHandlePending(boolean) @bci=54, line=191 (Compiled frame)
 - java.lang.ref.Reference$ReferenceHandler.run() @bci=1, line=153 (Interpreted frame)Locked ownable synchronizers:
    - None

Issue Links

• links to: GitHub Pull Request #2167