[RATIS-558] GrpcLogAppender doesn't reset the log entries to follower on reply to inconsistent entries

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 0.3.0
• Fix Version/s: 0.4.0
• Component/s: gRPC
• Labels: MiniOzoneChaosCluster
• Target Version/s: 0.4.0
• Assignee: Tsz-wo Sze
• Reporter: Mukul Kumar Singh
• Created: 19/May/19 07:16
• Updated: 23/May/19 06:43
• Resolved: 23/May/19 06:42

Description

Follower stuck in inconsistent state after node restart

2019-05-19 12:02:24,377 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(983)) - e143b976-ab35-4555-a800-7f05a2b1b738: Failed appendEntries as latest snapshot (5128) already h
as the append entries (first index: 0)
2019-05-19 12:02:24,378 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(965)) - e143b976-ab35-4555-a800-7f05a2b1b738: inconsistency entries. Reply:ad136ce5-92e0-4bd6-a302-cdd
4f8b7775b<-e143b976-ab35-4555-a800-7f05a2b1b738#34:FAIL,INCONSISTENCY,nextIndex:5129,term:4,followerCommit:5128
2019-05-19 12:02:24,383 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(983)) - e143b976-ab35-4555-a800-7f05a2b1b738: Failed appendEntries as latest snapshot (5128) already h
as the append entries (first index: 1)
2019-05-19 12:02:24,387 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(965)) - e143b976-ab35-4555-a800-7f05a2b1b738: inconsistency entries. Reply:ad136ce5-92e0-4bd6-a302-cdd4f8b7775b<-e143b976-ab35-4555-a800-7f05a2b1b738#35:FAIL,INCONSISTENCY,nextIndex:5129,term:4,followerCommit:5128
2019-05-19 12:02:24,387 ERROR server.GrpcLogAppender (GrpcLogAppender.java:onNext(236)) - Failed onNext request=ad136ce5-92e0-4bd6-a302-cdd4f8b7775b->e143b976-ab35-4555-a800-7f05a2b1b738#34-t4, previous=(t:0, i:0), leaderCommit=5182, initializing? false, entries: size=1, first=(t:1, i:0), CONFIGURATIONENTRY, reply=ad136ce5-92e0-4bd6-a302-cdd4f8b7775b<-e143b976-ab35-4555-a800-7f05a2b1b738#34:FAIL,INCONSISTENCY,nextIndex:5129,term:4,followerCommit:5128
java.lang.IllegalStateException
        at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:36)
        at org.apache.ratis.grpc.server.GrpcLogAppender.checkAndUpdateNextIndex(GrpcLogAppender.java:304)
        at org.apache.ratis.grpc.server.GrpcLogAppender.access$900(GrpcLogAppender.java:47)
        at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNextImpl(GrpcLogAppender.java:267)
        at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:234)
        at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:214)
        at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:421)
        at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
        at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
        at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519)
        at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
        at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at java.lang.Thread.run(Thread.java:748)

Issue Links

• incorporates: RATIS-557 IllegalStateException in Leader after Log Append