[RATIS-1583] Time / Temporal Based Ratis Snapshots

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: common
• Assignee: Unassigned
• Reporter: Nick Z
• Created: 17/May/22 18:35
• Updated: 10/Aug/22 22:50

Description

Currently Ratis only supports triggering snapshots after a certain number of transactions. However, in some use-cases it is more suitable to use a function of time rather than a specific number of transactions. This is especially true if the use-case is bursty or low-volume whereby snapshots may a specific number may not be met. In these cases it would actually be preferable to have a combination of time and / or a specific number of transactions. For example - trigger a snapshot if 60 seconds has passed or 1000 transactions have occurred. In fact Kafka has a similar concept when flushing a producers message buffer. This feature request would be especially useful in the context of Apache Ozone. See the conversation here about how delete operations are delayed when the number of Ratis transactions hasn't been met : https://the-asf.slack.com/archives/C5RK7PWA1/p1652234726073289?thread_ts=1652118090.322649&cid=C5RK7PWA1

Issue Links

• is related to: HDDS-7111 SCMHADBTransactionBuffer not flushed in time
• is required by: HDDS-6721 Ozone key deletion often dose not delete the real data blocks in datanode