[RATIS-1653] TestNettyDataStreamChainTopologyWithGrpcCluster fails sometimes

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 2.3.0
• Component/s: server, test

Environment

> java -version
openjdk version "1.8.0_332"
OpenJDK Runtime Environment (Temurin)(build 1.8.0_332-b09)
OpenJDK 64-Bit Server VM (Temurin)(build 25.332-b09, mixed mode)

> mvn -version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /home/maru/.sdkman/candidates/maven/current
Java version: 1.8.0_332, vendor: Temurin, runtime: /home/maru/.sdkman/candidates/java/8.0.332-tem/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "5.15.0-43-generic", arch: "amd64", family: "unix"

• Assignee: Unassigned
• Reporter: Kiyoshi Mizumaru
• Created: 01/Aug/22 10:30
• Updated: 02/Nov/23 04:53

Description

Sometimes I see mvn test fails with the following error:

[INFO] Running org.apache.ratis.datastream.TestNettyDataStreamChainTopologyWithGrpcCluster
[ERROR] Tests run: 4, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 94.492 s <<< FAILURE! - in org.apache.ratis.datastream.TestNettyDataStreamChainTopologyWithGrpcCluster
[ERROR] testMultipleStreamsMultipleServersStepDownLeader(org.apache.ratis.datastream.TestNettyDataStreamChainTopologyWithGrpcCluster)  Time elapsed: 63.959 s  <<< ERROR!
java.util.concurrent.CompletionException: org.apache.ratis.protocol.exceptions.TimeoutIOException: Timeout 10000ms: Failed to send DataStreamWindowRequest:seqNum=12,DataStreamRequestHeader:clientId=client-B7213C09F5FF,type=STREAM_DATA,id=418,offset=7575945,length=0
    at java.util.concurrent.CompletableFuture.encodeThrowable(CompletableFuture.java:292)
    at java.util.concurrent.CompletableFuture.completeThrowable(CompletableFuture.java:308)
    at java.util.concurrent.CompletableFuture.uniWhenComplete(CompletableFuture.java:783)
    at java.util.concurrent.CompletableFuture$UniWhenComplete.tryFire(CompletableFuture.java:750)
    at java.util.concurrent.CompletableFuture.postComplete(CompletableFuture.java:488)
    at java.util.concurrent.CompletableFuture.completeExceptionally(CompletableFuture.java:1990)
    at org.apache.ratis.client.impl.OrderedStreamAsync.lambda$scheduleWithTimeout$7(OrderedStreamAsync.java:172)
    at org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$0(TimeoutScheduler.java:141)
    at org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$1(TimeoutScheduler.java:155)
    at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:38)
    at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:79)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:750)
Caused by: org.apache.ratis.protocol.exceptions.TimeoutIOException: Timeout 10000ms: Failed to send DataStreamWindowRequest:seqNum=12,DataStreamRequestHeader:clientId=client-B7213C09F5FF,type=STREAM_DATA,id=418,offset=7575945,length=0
    ... 12 more 

I've uploaded output of TestNettyDataStreamChainTopologyWithGrpcCluster#testMultipleServersStepdownLeader test to my gist https://gist.github.com/kmizumar/4eefb95ac7677ab47442e3e17c920645