[IGNITE-22367] Create index metadata storage

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0-beta2
• Labels: ignite-3
• Epic Link: Catalog compaction support in indexes
• Assignee: Kirill Tkalenko
• Reporter: Roman Puchkovskiy
• Reviewer: Roman Puchkovskiy
• Created: 29/May/24 13:43
• Updated: 07/Jun/24 11:55
• Resolved: 07/Jun/24 11:55

Description

Currently, all index metadata is stored in the Catalog. This is problematic, because

1. Some fields (like txWaitCatalogVersion) or states (like STOPPING) do not belong to the Catalog
2. The Catalog may be compacted making it impossible to get metadata from it

We need to create a component (based on Vault) to store metadata for each index. An index entry can include (the list is rough):

1. Index ID
2. Name? (not sure if we need it; maybe for building messages for humans)
3. txWaitCatalogVersion
4. History of how the index changed its state (with activation timestamp of each transition)
5. Current state

The states set is broader than the set stored in the Catalog:

• REGISTERED
• BUILDING
• AVAILABLE
• STOPPED
• READ_ONLY (this means that the index was dropped (maybe together with its table), it cannot be written to, it can not be used by any RW transactions, but can be read by RO transactions with correct readTimestamp)
• DESTROYED? (it's not clear if it's needed or not)

The storage should provide efficient access to index metadata by index ID.

It should listen to Catalog updates and actualize itself if some changes happen to indexes.

If should also listen to LWM to switch from READ_ONLY state.

Issue Links

• blocks: IGNITE-21771 Use Index Metadata storage instead of Catalog
• links to: GitHub Pull Request #3887