[HDDS-9222] Write chunk cache blocking ratis for slow follower

• Type: Improvement
• Status: Reopened
• Priority: Major
• Resolution: Unresolved
• Component/s: Ozone Datanode
• Labels: proton
• Target Version/s: 1.5.0
• Assignee: Sumit Agrawal
• Reporter: Sumit Agrawal
• Created: 29/Aug/23 05:30
• Updated: 16/Dec/23 23:43

Description

writeChunk when try to add to cache is blocking, as cache might be full. This can happen,

• follower DN is very slow
• follower DN is down

Under this situation, close container and pipeline will be stuck for the ratis is its blocked.

 

As part of this, need write chunk being throttled, either,

• rejecting request
• OR going ahead without cache – Need handle parallel readChunk which can fail in this scenario (HDDS-2542 - handle this scenario with certain limitation)

 

Issue Links

• is related to: HDDS-1844 Tune the stateMachineDataCache to a reasonable fraction of Datanode Heap

Sub-Tasks

• Non-blocking container statemachine cache: Resolved
• Add metrics for stateMachine cache count and size in bytes: Open
• Follower cache to reduce cache miss in container state machine: Open
• Throttling strategy for write operation in container state machine: Open
• DN pipeline health monitor: Open
• unhealthy pipeline closure on slowness: Open