[RATIS-718] TimeoutScheduler throws IllegalStateException

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.5.0
• Component/s: util
• Assignee: Tsz-wo Sze
• Reporter: Shashikant Banerjee
• Created: 17/Oct/19 06:47
• Updated: 22/Oct/19 07:44
• Resolved: 22/Oct/19 07:44

Description

Shashikant Banerjee 10:38 AM2019-10-16 11:45:43,150 ERROR [java.util.concurrent.ThreadPoolExecutor$Worker@5d45ac64[State = -1, empty queue]] org.apache.ratis.client.impl.OrderedAsync: Failed* to retry RaftClientRequest:client-A67D4B37A25B->afda84a7-3b53-4b20-bd3b-5400b60a5014@group-DC6EDD5625BC, cid=9, seq=3*, RW, ...
java.lang.IllegalStateException
    at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:36)
    at org.apache.ratis.util.TimeoutScheduler.onTimeout(TimeoutScheduler.java:124)
    at org.apache.ratis.util.TimeoutScheduler.onTimeout(TimeoutScheduler.java:110)
    at org.apache.ratis.util.TimeoutScheduler.onTimeout(TimeoutScheduler.java:180)
    at org.apache.ratis.grpc.client.GrpcClientProtocolClient$AsyncStreamObservers.onNext(GrpcClientProtocolClient.java:318)
    at org.apache.ratis.grpc.client.GrpcClientRpc.sendRequestAsync(GrpcClientRpc.java:68)
    at org.apache.ratis.client.impl.OrderedAsync.sendRequest(OrderedAsync.java:208)
    at org.apache.ratis.client.impl.OrderedAsync.sendRequestWithRetry(OrderedAsync.java:169)
    at org.apache.ratis.util.SlidingWindow$Client.sendOrDelayRequest(SlidingWindow.java:278)
    at org.apache.ratis.util.SlidingWindow$Client.retry(SlidingWindow.java:294)
    at org.apache.ratis.client.impl.OrderedAsync.lambda$scheduleWithTimeout$7(OrderedAsync.java:195)
    at org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$0(TimeoutScheduler.java:113)
    at org.apache.ratis.util.TimeoutScheduler.lambda$onTimeout$1(TimeoutScheduler.java:133)
    at org.apache.ratis.util.LogUtils.runAndLog(LogUtils.java:50)
    at org.apache.ratis.util.LogUtils$1.run(LogUtils.java:91)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
    at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
    at java.lang.Thread.run(Thread.java:748)

Issue Links

• is broken by: RATIS-485 TimeoutScheduler is leaked by gRPC client implementation
• relates to: HDDS-2332 BlockOutputStream#waitOnFlushFutures blocks on putBlock combined future