ContainerStateMachine in DN have stateMachineDataCache, which cache data wrt to logIndex of ratis, this is used by leader to send data to other follower.
Issue: the cache gets cleared with incorrect logic, where all higher index is cleared, when apply transaction is called with lower index.
- with this clearing, when leader send data, it will cause disk read adding pressure over disk IO.
As solution, the check should be k <= (Math.min(minIndex, index)) where all previous index should be cleared as follower sync is done for that.
Impact with this change:
- cache is controlled using LeaderNumPendingRequests (write.element-limit) default 1024 and pendingRequestsBytesLimit (dfs.container.ratis.leader.pending.bytes.limit) default 1GB. So further cache will block till all follower gets sync. This will be correct controlling write load over DN till all cache in sync with majority of follower.