[HDDS-11366] Consider Replacing WriteBatch with WriteBatchWithIndex

• Type: New Feature
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: OM
• Assignee: Unassigned
• Reporter: Ivan Andika
• Created: 26/Aug/24 07:51
• Updated: 26/Aug/24 08:36

Description

Currently OM uses WriteBatch which only allows write on the batch. Therefore in OM Response, we should not call DB#get directly since the latest write for a particular key might be within the current batch that is yet to be flushed by the OM double buffer.

We might want to replace WriteBatch with WriteBatchIndex which allows us to read the latest writes from the write batch only (WriteBatchIndex#getFromBatch) or also with the DB (WriteBatchIndex#getFromBatchAndDB). 

Perhaps WriteBatchIndex remove the need of additional bookeeping in RDBBatchOperation#opCache by setting overWriteKey flag. However, we might still need to account for JNI call overhead of using WriteBatchIndex (multiple JNI calls) vs RDBBatchOperationo#opCache (single JNI call).

Related resources

https://www.cockroachlabs.com/blog/cockroachdb-on-rocksd/#blitzing-through-more-rocksdb-features#Indexed-Batches

https://github.com/facebook/rocksdb/wiki/Write-Batch-With-Index

https://rocksdb.org/blog/2015/02/27/write-batch-with-index.html

Issue Links

• relates to: HDDS-7483 RocksDB BatchWriteWithIndex to handle SCM buffer transaction
• relates to: HDDS-8128 Deduplicate the ops in RDBBatchOperation