[HDDS-9373] Optimize listKeysLight API further by efficient server-side proto field population

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: OM, Ozone Client
• Labels: blackbird pull-request-available
• Assignee: Tanvi Penumudy
• Reporter: Tanvi Penumudy
• Created: 03/Oct/23 06:18
• Updated: 08/Jan/24 08:36

Description

We have introduced a newer lightweight listKeys API - HDDS-9079. Implement a lightweight listKeys API (PR #5115) which utilizes a trimmed KeyInfo (BasicKeyInfo) proto object for excluding fields such as KeyLocations which are not very useful on the client side utilizing the server-side Ozone key encapsulations and proto messages newly introduced as part of HDDS-9079:

message BasicKeyInfo {
    optional string keyName = 1;
    optional uint64 dataSize = 2;
    optional uint64 creationTime = 3;
    optional uint64 modificationTime = 4;
    optional hadoop.hdds.ReplicationType type = 5;
    optional hadoop.hdds.ReplicationFactor factor = 6;
    optional hadoop.hdds.ECReplicationConfig ecReplicationConfig = 7;
}

The main objective is to be able to skip the deserialization of metadata specific to block information, etc for reducing the performance overhead for listing keys.

    optional uint64 modificationTime = 4;
    optional hadoop.hdds.ReplicationType type = 5;
    optional hadoop.hdds.ReplicationFactor factor = 6;
    optional hadoop.hdds.ECReplicationConfig ecReplicationConfig = 7;

We can improve the performance further by populating the above fields server-side only if the key's config differs from that of its bucket's config. This should reduce the payload and processing server-side unless the key is replicated in a config different from that of its bucket's default.

Issue Links

• is related to: HDDS-9079 Implement a lightweight listKeys API
• links to: GitHub Pull Request #5382