[HDDS-5384] OM refreshPipeline should not invoke the expensive OmKeyLocationInfoGroup.getLocationList()

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Implemented
• Fix Version/s: 1.2.0
• Component/s: OM
• Labels: pull-request-available
• Assignee: Wei-Chiu Chuang
• Reporter: Wei-Chiu Chuang
• Created: 24/Jun/21 15:14
• Updated: 13/Jul/21 13:40
• Resolved: 13/Jul/21 13:40

Description

The OM's refreshPipeline (used by liststatus) implementation iterates over OmKeyLocationInfoGroup.getLocationList(), which is a very expensive call. It iterate over a collection of list of objects, allocates a new list, perform operations on each of them. In short, it's an O( n ) method in terms of space and time complexity.

There are many places in the Ozone code that uses this method. Most usages iterates over the generated list, without modifying the list. We should instead return the collection of lists, which is O( 1 ).

I have a client that issues many listStatus calls to examine the effect. Before the change, refreshPipeline costs 8.65% of heap usage. After: 1.95%.
CPU cost: before: 8.18% after: 4.22%

We should refrain from invoking getLocationList() as much as possible. But given the wide usage in the code, I elect not to remove the usage completely to avoid destabilizing it. Instead, I changed the usage in refreshPipeline to demonstrate its impact.

Issue Links

• links to: GitHub Pull Request #2367