[HDDS-9766] Intermittent AlreadyClosedException in TestCommitWatcher.testReleaseBuffersOnException

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: test
• Labels: pull-request-available
• Assignee: Arafat Khan
• Reporter: Attila Doroszlai
• Created: 23/Nov/23 14:24
• Updated: 07/Dec/23 17:25
• Resolved: 29/Nov/23 20:24

Description

org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.testReleaseBuffersOnException -- Time elapsed: 28.88 s <<< ERROR!
java.util.concurrent.ExecutionException: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-2D6A59F17A72->RAFT is closed.
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.testReleaseBuffersOnException(TestCommitWatcher.java:301)
...
Caused by: org.apache.ratis.protocol.exceptions.AlreadyClosedException: SlidingWindow$Client:client-2D6A59F17A72->RAFT is closed.
	at org.apache.ratis.util.SlidingWindow$Client.alreadyClosed(SlidingWindow.java:406)
...
Caused by: org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-2D6A59F17A72->31ca1c78-6c1a-481a-9835-ed9e1bd72d7b@group-4121B0A26A9F, cid=40, seq=1*, Watch(0), null for 3 attempts with RequestTypeDependentRetryPolicy{WRITE->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@571a7a22, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->org.apache.ratis.retry.ExponentialBackoffRetry@571a7a22}), WATCH->ExceptionDependentRetry(maxAttempts=2147483647; defaultPolicy=MultipleLinearRandomRetry[5x5s, 5x10s, 5x15s, 5x20s, 5x25s, 10x60s]; map={org.apache.ratis.protocol.exceptions.GroupMismatchException->NoRetry, org.apache.ratis.protocol.exceptions.NotReplicatedException->NoRetry, org.apache.ratis.protocol.exceptions.ResourceUnavailableException->org.apache.ratis.retry.ExponentialBackoffRetry@571a7a22, org.apache.ratis.protocol.exceptions.StateMachineException->NoRetry, org.apache.ratis.protocol.exceptions.TimeoutIOException->NoRetry})}
	at org.apache.ratis.client.impl.RaftClientImpl.noMoreRetries(RaftClientImpl.java:353)
	... 20 more
Caused by: org.apache.ratis.protocol.exceptions.NotLeaderException: Server 31ca1c78-6c1a-481a-9835-ed9e1bd72d7b@group-4121B0A26A9F is not the leader 66f0d3a6-9de9-4155-af63-8575b18d2e1d|10.1.0.11:15121
	at org.apache.ratis.client.impl.ClientProtoUtils.toRaftClientReply(ClientProtoUtils.java:397)
	at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers$1.onNext(GrpcClientProtocolClient.java:310)
	... 11 more

• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/10/19/26151/it-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.txt
• https://github.com/adoroszlai/ozone-build-results/blob/master/2023/11/23/26994/it-hdds/hadoop-ozone/integration-test/org.apache.hadoop.hdds.scm.storage.TestCommitWatcher.txt

Issue Links

• Testing discovered: HDDS-9805 Reduce cluster size in integration tests
• links to: GitHub Pull Request #5700