[HDDS-3630] Merge rocksdb in datanode

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Labels: pull-request-available
• Assignee: Mark Gui
• Reporter: runzhiwang
• Created: 20/May/20 12:18
• Updated: 30/Jan/23 18:54
• Resolved: 07/Jun/22 08:48

Description

Currently, one rocksdb for one container. one container has 5GB capacity. 10TB data need more than 2000 rocksdb in one datanode. It's difficult to limit the memory of 2000 rocksdb. So maybe we should limited instance of rocksdb for each disk.

The design of improvement is in the follow link, but still is a draft.
TODO:
1. compatibility with current logic i.e. one rocksdb for each container
2. measure the memory usage before and after improvement
3. effect on efficiency of read and write.

https://docs.google.com/document/d/18Ybg-NjyU602c-MYXaJHP6yrg-dVMZKGyoK5C_pp1mM/edit#

 

Updated: 2021-10-29

We'll continue this topic with a new detailed design doc:

https://docs.google.com/document/d/1hLtPJI8lSUhazMsoRKQuXuXs2gqixi6bQNTV9UnPsqs/edit?usp=sharing

 

Updated: 2022-05-23

Refined the design doc:

https://docs.google.com/document/d/1oPCRUVqTJDbYITbwvoVCOZ2-k108FwOXVhq9ukDKHbQ/edit?usp=sharing

Issue Links

• fixes: HDDS-2317 Change rocksDB per Container model to have table per container on RocksDb per disk
• fixes: HDDS-3267 Replace ContainerCache in BlockUtils by LoadingCache
• links to: GitHub Pull Request #3331
• links to: GitHub Pull Request #3469
• links to: GitHub Pull Request #3470
• links to: GitHub Pull Request #3471
• links to: GitHub Pull Request #3474

Sub-Tasks

• Allow to add ColumnFamilyOptions to tables.: Resolved
• Decouple DatanodeStore impl classes from container.: Resolved
• Add DBHandle abstraction over ReferenceCountedDB.: Resolved
• Format table key according to schema in KeyValueContainerData.: Resolved
• Add prefix iterator support to RDBTable.: Resolved
• Add new container schema v3 definitions.: Resolved
• Add a Cache for per-disk DB handles.: Resolved
• Per-disk DB location management.: Resolved
• KeyValueContainer operation adaptation for schema v3 containers.: Resolved
• BlockDeletingService adaptation for schema v3 containers.: Resolved
• New container replication operations for schema v3 container.: Resolved
• Non-rolling upgrade supports container schema V3: Resolved
• Add scanner for dbVolume: Resolved
• Create rocksdb instance for hddsVolume when schema3 is finalized: Resolved
• Flush RocksDB WAL when closing container in Schema V3: Resolved
• Persist RocksDB path of each HddsVolume in VERSION file: Open
• Support configurable separator between container ID and container meta data key: Resolved
• Add compatibility test between schema v3 & v2.: Resolved
• Add document for Merge RocksDB in datanode feature: Resolved
• Add smoke test for Merge RocksDB in datanode feature: Resolved
• Fix issues reportd by sonar: Resolved
• ldb cli command supports to scan container V3: Resolved
• Ozone debug container CLI supports container export: Resolved
• Add property to control RocksDB max open files: Resolved
• Expose RocksDB critical properties: Resolved
• Add support for Rocksdb user info LOG configure: Resolved
• Schema V3 RocksDB instance statitics register as metrics collision: Resolved
• Enable Container SchemaV3 by default: Resolved
• Auto rocksDB small sst files compaction: Resolved
• Test container schema V2 and V3 replica mutual import: Resolved
• Add container layout v2 upgrade to v3 tool: Resolved
• Add document for container layout v2 upgrade to v3 tool: Open
• Add container layout v2 upgrade to v3 logic: Resolved
• Update documentation to mention that container schemaV3 is default: Resolved