[HDDS-7321] Auto rocksDB small sst files compaction

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.4.0
• Labels: pull-request-available
• Assignee: Sammi Chen
• Reporter: Sammi Chen
• Created: 13/Oct/22 07:43
• Updated: 07/Jul/23 13:33
• Resolved: 08/Feb/23 06:13

Description

RocksDB has auto compaction itself, which is triggered by the total level file size. 

Once the total level file size exceeds the threshold, RocksDB will schedule the compaction in backend.

 

When replicating containers between datanodes, current implementation leverages RocksDB  SstFileWriter to export container meta data to individual sst files, and leverages RocksDB ingestExternalFile to import container meta data sst files directly into target datanode RocksDB. If the imported container meta data keys don't overlap with other sst files(Consider Merge RocksDB design, container ID is used as prefix of each meta data key,  this is true for most of time), the imported sst file will be kept and remain the same without compacting with other existing sst files.

The worst case, if thousands or dozens of thousands of containers are imported on one datanode, there would be dozens of thousands of small sst files under one RocksDB, or across all RocksDB instances of one datanode.  By default,  RocksDB has no limit of open files.  Dozens of thousands of small sst files would exhaust the process open file quota, and service stability will be impacted.

This task aims to provide a way to auto compact all these small sst files into merged big ones.  Of course, the compaction will have impact on user data read/write performance on this datanode.

Issue Links

• relates to: HDDS-8980 RocksDB crash in RocksDBStoreMetrics.computeSstFileStat
• links to: GitHub Pull Request #3908