[HDDS-7111] SCMHADBTransactionBuffer not flushed in time

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Fix Version/s: HDDS-7483
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Xu Shao Hong
• Created: 10/Aug/22 03:03
• Updated: 25/Apr/23 02:18
• Resolved: 22/Feb/23 23:51

Description

We recently found that after we deleted all the keys, the cluster will remain with some data. One of the reasons is due to this:

SCM will add deleted blocks into transactions when receiving the request from OM. when HA is enabled,  DBTransactionbuffer is implemented as the SCMHADBTransactionbufferImpl. Inside this, the buffer will not be flushed immediately. Normally, it will be flushed when SCM takes a snapshot. The snapshot gap threshold is default 1000. If the user has little load pressure on the cluster (no writing more ratis logs), the buffer will be always pending in the memory. Real deletion happened in SCMBlockDeletingService, which will scan the DB and get the transactions, for those txns in-memory buffer, it won't find them.

This is why DN has not yet received these deleted block info.

We have three solutions:

1. add a global view for all the deleted block transactions, but it will bring too much pressure on the memory.
2. use `SCMDBTransactionBufferImpl` instead of SCMHADBTransactionBufferImpl in `DeletedBlockLog`. It could work but may be confusing. Why does it need the direst flush independently?
3. Add a flush monitor/ check to trigger the non-empty flush, overall, this could be a common mechanism if some other cases like the deleted blocks appears in the future.

Issue Links

• is related to: HDDS-7483 RocksDB BatchWriteWithIndex to handle SCM buffer transaction
• relates to: HDDS-6721 Ozone key deletion often dose not delete the real data blocks in datanode
• relates to: RATIS-1583 Time / Temporal Based Ratis Snapshots
• links to: GitHub Pull Request #3670