[RATIS-1836] Reduce retry frequency by leader on failure path for appendEntry

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0, 2.5.2
• Component/s: gRPC
• Labels: pull-request-available
• Assignee: Sumit Agrawal
• Reporter: Sumit Agrawal
• Created: 17/Apr/23 16:13
• Updated: 30/Nov/23 17:16
• Resolved: 02/Jun/23 07:52

Description

During test of ozone make one of DN down, its observed,

1. HeartBeat timeout and retry every second
2. Close ratis pipeline, it retry every 10 millisecond even follower node not present

On Failure path, there should not be aggressive retry,

 

Performance Impact on Leader DN where one of DN is down, this will overload the DN with very frequent retry and pilling of netty objects.

Issue Links

• causes: HDDS-8299 Disk full situation on a leader DN may result in followers getting stuck in a retry loop
• is duplicated by: HDDS-8297 Do not close open pipelines and open containers when a node becomes stale
• relates to: RATIS-1945 INCONSISTENCY replies should be counted as errors.
• links to: GitHub Pull Request #883