[HDDS-10788] Intermittent failure in testWatchForCommitForRetryfailure

• Type: Sub-task
• Status: Patch Available
• Priority: Major
• Resolution: Unresolved
• Component/s: test
• Labels: pull-request-available
• Target Version/s: 2.0.0
• Assignee: Raju Balpande
• Reporter: Attila Doroszlai
• Created: 02/May/24 06:32
• Updated: 13/Oct/24 17:23

Description

Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 134.993 s <<< FAILURE! - in org.apache.hadoop.ozone.client.rpc.TestWatchForCommit
org.apache.hadoop.ozone.client.rpc.TestWatchForCommit.testWatchForCommitForRetryfailure  Time elapsed: 37.042 s  <<< FAILURE!
java.lang.AssertionError: 

Expecting:
 <"org.apache.ratis.protocol.exceptions.RaftRetryFailureException: Failed RaftClientRequest:client-35B425B0E6BC->7ca6ef5b-8396-458a-a334-21f1f3211157@group-3A5CD4773BBD, cid=51, seq=null, Watch-MAJORITY_COMMITTED(95), null for 3 attempts with RequestTypeDependentRetryPolicy{...}">
not to contain:
 <"Watch-MAJORITY_COMMITTED">

	at org.apache.hadoop.ozone.client.rpc.TestWatchForCommit.testWatchForCommitForRetryfailure(TestWatchForCommit.java:295)

https://github.com/apache/ozone/blob/a658802d628271efa82824dc3c316f8eebfc75d3/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestWatchForCommit.java#L292-L296

      // client should not attempt to watch with
      // MAJORITY_COMMITTED replication level, except the grpc IO issue
      if (!logCapturer.getOutput().contains("Connection refused")) {
        assertThat(e.getMessage()).doesNotContain("Watch-MAJORITY_COMMITTED");
      }

log

[main] WARN  scm.XceiverClientRatis (XceiverClientRatis.java:watchForCommit(284)) - 3 way commit failed on pipeline ...
java.util.concurrent.ExecutionException: org.apache.ratis.protocol.exceptions.NotReplicatedException: Request with call Id 50 and log index 95 is not yet replicated to ALL_COMMITTED
	at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
	at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
	at org.apache.hadoop.hdds.scm.XceiverClientRatis.watchForCommit(XceiverClientRatis.java:279)
	at org.apache.hadoop.ozone.client.rpc.TestWatchForCommit.lambda$testWatchForCommitForRetryfailure$0(TestWatchForCommit.java:286)
	at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:53)
	at org.junit.jupiter.api.AssertThrows.assertThrows(AssertThrows.java:35)
	at org.junit.jupiter.api.Assertions.assertThrows(Assertions.java:3115)
	at org.apache.hadoop.ozone.client.rpc.TestWatchForCommit.testWatchForCommitForRetryfailure(TestWatchForCommit.java:285)

Issue Links

• relates to: HDDS-9826 Fix exception handling if one Datanode is not available (Ratis)
• links to: GitHub Pull Request #6797