[HDDS-11660] Recon List Key API: Reduce object creation and buffering memory

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 2.0.0
• Component/s: Ozone Recon
• Labels: pull-request-available
• Assignee: Stephen O'Donnell Assign to me
• Reporter: Stephen O'Donnell
• Created: 07/Nov/24 13:23
• Updated: 08/Nov/24 10:01
• Resolved: 08/Nov/24 10:01

Description

The current ListKeysAPI implementation, pulls a batch of OMKeyInfo objects from the File / Key table and buffers them in an array.

Then it converts each item in that array to a KeyEntityInfo object, which has JSON annotations allowing the data to be serialized over HTTP.

This results in two copies of the objects buffered in memory at once.

Additionally, the OMKeyInfo object is a fairly heavyweight object, and creating it from protobuf also deserialized the block information, which is not needed for the ListKey purposes.

Further taking the protobuffer bytes from RocksDB creates a KeyInfo object (deserialized proto) and then this object is copied into OmKeyInfo.

So for each key retrieved, we start with:

1. A deserialized proto object (KeyInfo)
2. A copy of the proto fields (OmKeyInfo)
3. A json annotated object (KeyEntityInfo).

By deserializing into the 3rd object directly, and having it wrap a keyInfo object and delegate calls to it when appropriate, we avoid needing the heavyweight OmKeyInfo object, and make the KeyEntityInfo object "slimmer". We can also avoid the double buffering, creating only a single list of objects for the batch.

This PR introduces a KeyEntityInfoProtoWrapper object, rather than changing the existing KeyEntityInfo object. The reason, is that other APIs outside of ListKeys use it. I am not ready to change those APIs now, however after this change gets committed and we are happy with it, it should be possible to apply similar changes to those other API and eventually remove KeyEntityInfo from the system.

Issue Links

• links to: GitHub Pull Request #7405