[RATIS-1790] Improve gRPC LogAppender's deadline mechanism

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 2.4.1
• Fix Version/s: 3.0.0
• Component/s: gRPC, snapshot
• Assignee: Tsz-wo Sze
• Reporter: Song Ziyang
• Created: 23/Feb/23 08:36
• Updated: 14/Oct/23 08:05
• Resolved: 14/Oct/23 08:05

Description

1. Issues related to gRPC logAppender

1. (100% reproduce) gRPC appender will timeout and fail when installing a large snapshot to follower, as previously reported in https://issues.apache.org/jira/browse/RATIS-1782.
2. (small probability) Storms of inconsistent RPCs bouncing between leader and followers, as previously reported in https://issues.apache.org/jira/browse/RATIS-1674.

2. Cause of these issues

Current deadline configuration of gRPC bidirectional streaming leads to the issues above.

3. Dive into gRPC logAppender

gRPC logAppender will generate the stub with a deadline at the beginning of installSnapshot, as in [1] and [2].

snapshotRequestObserver = getClient().installSnapshot(responseHandler);
for (InstallSnapshotRequestProto request : newInstallSnapshotRequests(requestId, snapshot)) {
    snapshotRequestObserver.onNext(request);
}

Notice that the deadline is set for the whole observer, not for each streaming message. Deadline is a fixed time point in future, every streaming messages should complete before this time point, otherwise will be cancelled and onError will be invoked.
I guess the original implementor of installSnapshot treats deadline the same as timeout, which they are not (check https://grpc.io/blog/deadlines/ for their difference). Therefore, every streaming messages will not have a independent timeout of 3s (which we want), but rather share the same deadline of (initial_time + 3s). When snapshot is large, the RPCs ordered lately will become timeout and fail. This is the cause for 1st issue I mentioned above. Also, gRPC implementors does not recommend to use deadline in a streaming stub (see https://github.com/grpc/grpc-java/issues/5498#issuecomment-476299936)

AppendEntries is not affected by this deadline problem since it does not assign a deadline to stub [3]. However, not using a deadline also causes some unexpected behaviors, as mentioned in 2nd issue. Every RPC submitted to appendEntries observer will never timeout and will guaranteed to be delivered to the follower. There are max 128 pending requests in gRPC's sending queue. Consider, if the first pending-RPC receives an inconsistent reply, we shall cancel every other RPCs in this sending queue. However, these submitted RPCs are un-cancellable without a deadline, all we can do is to see them being sent, helplessly. These sent requests can cause inconsistency storms, refer to RATIS-1674.

Issue Links

• is related to: RATIS-1807 Support timeout in gRPC
• is related to: RATIS-1809 Separate gRPC LogAppender unary / streaming request timeout
• is related to: RATIS-1812 Enforce a outstanding request limit in StreamObserverWithTimeout
• links to: GitHub Pull Request #828
• links to: GitHub Pull Request #842