[HDDS-5332] Add a new column family and a service provider in Recon DB for Namespace Summaries

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Labels: pull-request-available
• Assignee: Yuan Gu
• Reporter: Siyao Meng
• Created: 10/Jun/21 18:00
• Updated: 29/Apr/22 00:04
• Resolved: 06/Jul/21 16:50

Description

Table schema:

objectId --> class NSSummary

New class NSSummary should at least include these fields:

1. Total number of files directly[1] under this directory,
2. Total size of files directly under this directory,
3. File Size Buckets of files directly under this directory.

Note: [1] directly means not counting files under any subdirectories, for now. We could further optimize the speed later by trading more writes (propagating the statistics any layers deep upwards) with speed. This is for reducing potential write-amplification for now. The stats could be extended into arbitrary layers deep later.

[06/21] Renamed ReconContainerDBProvider to ReconDBProvider

[06/21] Expanded the scope of this issue by designing a service provider interface for namespace summary, which wraps up all operations on the new schema.

 

[06/23] Merge with HDDS-5379

Because we are intending to add a new CF to the Recon container DB, the existing ContainerDBServiceProvider must be refactored to resolve conflict with a new service provider sharing the same RDB as the current one directly operates on the DB not just the CF/tables.

1. Refactor ContainerDBServiceProvider into something not directly related to the underlying RDB
2. Use a new ReconDBStore instead of containerDbStore in ContainerDBServiceProviderImpl
3. Rename ContainerDBServiceProvider/ContainerDBServiceProviderImpl into sth more appropriate
4. Combined ReconRocksDB and ReconDBProvider into one class, which manages all db-level operations. (Retained ReconRocksDB, deleted ReconDBProvider, fixed injection dependency in ReconControllerModule)

Issue Links

• fixes: HDDS-6667 Recon can crash if processing a container report after installing an OM snapshot
• links to: GitHub Pull Request #2366