[RATIS-2185] Improve gRPC log messages debugability

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: gRPC
• Assignee: Unassigned
• Reporter: Duong
• Created: 02/Nov/24 01:20
• Updated: 15 hours ago

Description

Created this as a umbrella ticket for all changes related to improve the debugability of Ratis logs.

Today, there're a couple of issues with ratis logs making debug issues in ratis a hard job. For example:

• Lack of details: when an RPC call, e.g. appendEntries calls, fail, there is very little information to identify the problem, exception some vague message like "Unavailable" or "Client cancelled".  We've seen examples where "Client cancelled" is displayed for technical problem in GRPC layer, e.g. when appendEntries buffers are bigger than the max GRPC acceptable message size.
• Repetitive: TBF

Issue Links

• relates to: RATIS-2135 The leader keeps sending inconsistent entries repeatedly to followers.
• links to: GitHub Pull Request #1184