Uploaded image for project: 'Ratis'
  1. Ratis
  2. RATIS-1790

Improve gRPC LogAppender's deadline mechanism

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 2.4.1
    • 3.0.0
    • gRPC, snapshot
    • None

    Description

      1. Issues related to gRPC logAppender

      1. (100% reproduce) gRPC appender will timeout and fail when installing a large snapshot to follower, as previously reported in https://issues.apache.org/jira/browse/RATIS-1782.
      2. (small probability) Storms of inconsistent RPCs bouncing between leader and followers, as previously reported in https://issues.apache.org/jira/browse/RATIS-1674.

      2. Cause of these issues

      Current deadline configuration of gRPC bidirectional streaming leads to the issues above.

      3. Dive into gRPC logAppender

      gRPC logAppender will generate the stub with a deadline at the beginning of installSnapshot, as in [1] and [2].

      snapshotRequestObserver = getClient().installSnapshot(responseHandler);
      for (InstallSnapshotRequestProto request : newInstallSnapshotRequests(requestId, snapshot)) {
          snapshotRequestObserver.onNext(request);
      }
      

      Notice that the deadline is set for the whole observer, not for each streaming message. Deadline is a fixed time point in future, every streaming messages should complete before this time point, otherwise will be cancelled and onError will be invoked.
      I guess the original implementor of installSnapshot treats deadline the same as timeout, which they are not (check https://grpc.io/blog/deadlines/ for their difference). Therefore, every streaming messages will not have a independent timeout of 3s (which we want), but rather share the same deadline of (initial_time + 3s). When snapshot is large, the RPCs ordered lately will become timeout and fail. This is the cause for 1st issue I mentioned above. Also, gRPC implementors does not recommend to use deadline in a streaming stub (see https://github.com/grpc/grpc-java/issues/5498#issuecomment-476299936)

      AppendEntries is not affected by this deadline problem since it does not assign a deadline to stub [3]. However, not using a deadline also causes some unexpected behaviors, as mentioned in 2nd issue. Every RPC submitted to appendEntries observer will never timeout and will guaranteed to be delivered to the follower. There are max 128 pending requests in gRPC's sending queue. Consider, if the first pending-RPC receives an inconsistent reply, we shall cancel every other RPCs in this sending queue. However, these submitted RPCs are un-cancellable without a deadline, all we can do is to see them being sent, helplessly. These sent requests can cause inconsistency storms, refer to RATIS-1674.

      Attachments

        1. 828_test_passed.patch
          26 kB
          Song Ziyang
        2. 828_test.patch
          18 kB
          Tsz-wo Sze
        3. 828_test2.patch
          21 kB
          Tsz-wo Sze

        Issue Links

          Activity

            People

              szetszwo Tsz-wo Sze
              William Song Song Ziyang
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Time Tracking

                  Estimated:
                  Original Estimate - Not Specified
                  Not Specified
                  Remaining:
                  Remaining Estimate - 0h
                  0h
                  Logged:
                  Time Spent - 3.5h
                  3.5h