[RATIS-2077] Timedout StateMachine retainRead is released twice.

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.1.0
• Assignee: Duong
• Reporter: Duong
• Created: 07/May/24 16:07
• Updated: 11/May/24 21:15
• Resolved: 11/May/24 21:15

Description

When getEntry gets timed out, the StateMachine reference count is released twice.

java.util.concurrent.TimeoutException
        at java.base/java.util.concurrent.CompletableFuture.timedGet(CompletableFuture.java:1886)
        at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2021)
        at org.apache.ratis.server.raftlog.RaftLogBase$EntryWithDataImpl.getEntry(RaftLogBase.java:497)
        at org.apache.ratis.util.DataQueue.pollList(DataQueue.java:137)
        at org.apache.ratis.server.leader.LogAppenderBase.nextAppendEntriesRequest(LogAppenderBase.java:274)
        at org.apache.ratis.grpc.server.GrpcLogAppender.appendLog(GrpcLogAppender.java:387)
        at org.apache.ratis.grpc.server.GrpcLogAppender.run(GrpcLogAppender.java:262)
        at org.apache.ratis.server.leader.LogAppenderDaemon.run(LogAppenderDaemon.java:80)
        at java.base/java.lang.Thread.run(Thread.java:834)

 

it results in a ISE:

java.lang.IllegalStateException: Failed to release: object has already been completely released.
        at org.apache.ratis.util.ReferenceCountedLeakDetector$Impl.release(ReferenceCountedLeakDetector.java:130)
        at org.apache.ratis.util.ReferenceCountedLeakDetector$AdvancedTracing.release(ReferenceCountedLeakDetector.java:203)
        at org.apache.ratis.server.raftlog.RaftLogBase$1.lambda$updateFuture$0(RaftLogBase.java:439)
        at java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
        at java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
        at java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
        at java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
        at org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$retainRead$8(ContainerStateMachine.java:846)
        at java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:834) 

Object is already released at:
org.apache.ratis.server.raftlog.RaftLogBase$EntryWithDataImpl.lambda$discardData$0(RaftLogBase.java:528)
java.base/java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:859)
java.base/java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:837)
java.base/java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:506)
java.base/java.util.concurrent.CompletableFuture.complete(CompletableFuture.java:2073)
org.apache.hadoop.ozone.container.common.transport.server.ratis.ContainerStateMachine.lambda$retainRead$8(ContainerStateMachine.java:846)
java.base/java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1736)
java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
java.base/java.lang.Thread.run(Thread.java:834)

 

Issue Links

• links to: GitHub Pull Request #1081