Uploaded image for project: 'Hadoop Distributed Data Store'
  1. Hadoop Distributed Data Store
  2. HDDS-3227

Ensure eviction of stateMachineData from cache only when both followers catch up

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Ozone Datanode
    • Labels:
    • Target Version/s:

      Description

      Currently, the data in the StateMachineCache is evicted as soon as the applyTransaction call is issued for a transaction in Ratis. In our testing with keys in few kbs of size, it was figured that the data is evicted from the cache before append requests can be processed in a slightly slow follower thereby making leader read the chunk data from underlying fs/disk very frequently. This leads to slowing down the leader as well as well as overall throughput of the pipeline. 

      The idea here is to ensure the data is evicted from the cache only when both followers have caught up with the match index. If a follower is really slow, it will eventually be marked slow after nodeFailureTimeout and pipeline will be destroyed.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                shashikant Shashikant Banerjee
                Reporter:
                shashikant Shashikant Banerjee
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated: