Uploaded image for project: 'Ratis'
  1. Ratis
  2. RATIS-558

GrpcLogAppender doesn't reset the log entries to follower on reply to inconsistent entries

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 0.3.0
    • Fix Version/s: 0.4.0
    • Component/s: gRPC
    • Target Version/s:

      Description

      Follower stuck in inconsistent state after node restart

      2019-05-19 12:02:24,377 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(983)) - e143b976-ab35-4555-a800-7f05a2b1b738: Failed appendEntries as latest snapshot (5128) already h
      as the append entries (first index: 0)
      2019-05-19 12:02:24,378 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(965)) - e143b976-ab35-4555-a800-7f05a2b1b738: inconsistency entries. Reply:ad136ce5-92e0-4bd6-a302-cdd
      4f8b7775b<-e143b976-ab35-4555-a800-7f05a2b1b738#34:FAIL,INCONSISTENCY,nextIndex:5129,term:4,followerCommit:5128
      2019-05-19 12:02:24,383 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(983)) - e143b976-ab35-4555-a800-7f05a2b1b738: Failed appendEntries as latest snapshot (5128) already h
      as the append entries (first index: 1)
      2019-05-19 12:02:24,387 INFO  impl.RaftServerImpl (RaftServerImpl.java:checkInconsistentAppendEntries(965)) - e143b976-ab35-4555-a800-7f05a2b1b738: inconsistency entries. Reply:ad136ce5-92e0-4bd6-a302-cdd4f8b7775b<-e143b976-ab35-4555-a800-7f05a2b1b738#35:FAIL,INCONSISTENCY,nextIndex:5129,term:4,followerCommit:5128
      2019-05-19 12:02:24,387 ERROR server.GrpcLogAppender (GrpcLogAppender.java:onNext(236)) - Failed onNext request=ad136ce5-92e0-4bd6-a302-cdd4f8b7775b->e143b976-ab35-4555-a800-7f05a2b1b738#34-t4, previous=(t:0, i:0), leaderCommit=5182, initializing? false, entries: size=1, first=(t:1, i:0), CONFIGURATIONENTRY, reply=ad136ce5-92e0-4bd6-a302-cdd4f8b7775b<-e143b976-ab35-4555-a800-7f05a2b1b738#34:FAIL,INCONSISTENCY,nextIndex:5129,term:4,followerCommit:5128
      java.lang.IllegalStateException
              at org.apache.ratis.util.Preconditions.assertTrue(Preconditions.java:36)
              at org.apache.ratis.grpc.server.GrpcLogAppender.checkAndUpdateNextIndex(GrpcLogAppender.java:304)
              at org.apache.ratis.grpc.server.GrpcLogAppender.access$900(GrpcLogAppender.java:47)
              at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNextImpl(GrpcLogAppender.java:267)
              at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:234)
              at org.apache.ratis.grpc.server.GrpcLogAppender$AppendLogResponseHandler.onNext(GrpcLogAppender.java:214)
              at org.apache.ratis.thirdparty.io.grpc.stub.ClientCalls$StreamObserverToCallListenerAdapter.onMessage(ClientCalls.java:421)
              at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
              at org.apache.ratis.thirdparty.io.grpc.ForwardingClientCallListener.onMessage(ForwardingClientCallListener.java:33)
              at org.apache.ratis.thirdparty.io.grpc.internal.ClientCallImpl$ClientStreamListenerImpl$1MessagesAvailable.runInContext(ClientCallImpl.java:519)
              at org.apache.ratis.thirdparty.io.grpc.internal.ContextRunnable.run(ContextRunnable.java:37)
              at org.apache.ratis.thirdparty.io.grpc.internal.SerializingExecutor.run(SerializingExecutor.java:123)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
              at java.lang.Thread.run(Thread.java:748)
      

        Attachments

        1. r558_20190522b.patch
          2 kB
          Tsz-wo Sze
        2. r558_20190522.patch
          21 kB
          Tsz-wo Sze
        3. r558_20190520.patch
          16 kB
          Tsz-wo Sze

          Issue Links

            Activity

              People

              • Assignee:
                szetszwo Tsz-wo Sze
                Reporter:
                msingh Mukul Kumar Singh
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: