Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
This jira will track all changes required for introducing a separate prefix for keys in the OM.
In the current implementation, OM stores every key in a Key table, which maps the full key to a KeyInfo.
FS semantics require unique names. Specifically - inside any given directory (the prefix of any OM object key), each object must have a unique name. This means that a sub-directory and a file which share the same prefix, cannot have the same name.
if there exists an object(file) "/a/b/c/d/e/file" in OM, we cannot allow a new object(file/dir) "/a/b/c" to be created. it is implicit as per FS semantics that /a/b/c is the name of an existing directory.
The problem is, OM stores full keys in the key table. So the file create handler must iterate through the entire key table to determine uniqueness of the name. This makes create operations very resource intensive.
To improve this behaviour, we propose to store every valid prefix in a separate table in the OM, alongside the Key Table. The File create request handler can iterate this KeyPrefix table to check if the name already exists. Specific optimisations will be covered as separate jira items.
The following 2 sets of tasks are required to complete the key prefix infrastructure - base infra followed by extension to achieve prefix compression(in-memory + on-disk) for the OM Key table.
Base Infra:
1. Create prefix entry in KeyPrefix OM Table for every new key inserted in the OM Key Table - include file create, directory create and rename. Change File create request handler to check the KeyPrefix table, instead of iterating the entire Key Table.
2. Handle stale prefix entries in the KeyPrefix table - handle the situation where the last object associated with a prefix gets deleted. This task involves defining the semantics in this situation and subsequently the design for detecting stale entries.
Extension to support prefix compression in Key Table:
3. Assign unique ID to each prefix in the KeyPrefix table - include synchronisation.
4. Node restart - ensure the unique ID generation scheme handles restart.
5. Change the key in OM Key Table to <Prefix ID>/<name> instead of the full name.