[HDDS-8128] Deduplicate the ops in RDBBatchOperation

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Component/s: db
• Labels: pull-request-available
• Target Version/s: 1.3.1
• Assignee: Tsz-wo Sze
• Reporter: Tsz-wo Sze
• Created: 09/Mar/23 22:04
• Updated: 26/Aug/24 07:52
• Resolved: 25/Mar/23 15:03

Description

In a multipart upload test, the key "testKey" had 1000-parts with 8KB each. The same key was uploaded 10 times sequentially (i.e. it overwrote the previous upload) in a newly formatted cluster. The replication was 3, so the total raw size of the key is ~ 24 MB. After the test has completed, OM rocks db uses ~ 7.5 GB.

In this JIRA, we add a cache to RDBBatchOperation for deduplication. Within a batch, the put-ops and delete-ops of the same key can be safely deduplicated. Only the last op has to be applied to the db. All the previous ops can be discarded.

Issue Links

• is related to: HDDS-11366 Consider Replacing WriteBatch with WriteBatchWithIndex
• is related to: HDDS-8238 Multipart upload generates a huge OM rocksdb log
• links to: GitHub Pull Request #4424