[HDDS-9272] Performance improvement for OM's sort datanodes

• Type: Improvement
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Labels: ozone-performance pull-request-available
• Assignee: Tanvi Penumudy
• Reporter: Duong
• Created: 12/Sep/23 16:52
• Updated: 05/Oct/23 03:26

Description

Problem

After HDDS-8300, get key metadata API in OM sorts datanodes to allow clients to optimize data I/O based on locality. However, today this requires OM to make additional calls to SCM to sort the datanodes. These calls contribute more than 80% of the getKeyInfo latency and reduce the peak pure read OPPS from 100K to 25K. 

Optimization

Today, the get key metadata API makes 2 separate calls to SCM, one to get container pipelines and the second to sort datanodes in the container pipelines (see KeyManagerImpl#getKeyInfo): 

    if (!args.isHeadOp()) {
    ...
      // get container pipeline info from cache.
      captureLatencyNs(metrics.getGetKeyInfoRefreshLocationLatencyNs(),
          () -> refreshPipelineFromCache(value,
              args.isForceUpdateContainerCacheFromSCM()));

      if (args.getSortDatanodes()) {
        sortDatanodes(clientAddress, value);
      }

The two calls can be combined into one and the result is cached in the existing containerLocationCache in ScmClient.
Some details of the implementation:

1. A new SCM API, called `Pipeline getContainerPipeline(long containerId, boolean sortDatanodes)` is added to `SCMClientProtocolServer`. When the second argument is true, the datanodes in the result pipeline are sorted by the logic in `SCMBlockProtocolServer#sortDatanodes`. Another API `Map<Long, Pipeline> getContainerPipelineBatch(List<Long> containerIds, boolean sortDatanodes)` is also added for batch calls. It's important to implement the changes in new API leave existing SCM API untouched for compatibility.
2. In `org.apache.hadoop.ozone.om.ScmClient`, the read-through `containerLocationCache` now uses the two new APIs to load container pipelines instead of `getContainerWithPipeline` and `getContainerWithPipelineBatch`. See `org.apache.hadoop.ozone.om.ScmClient#createContainerLocationCache`. The cache key of `containerLocationCache` is now a composition `PipelineCacheKey(long containerId, boolean datanodeSorted)`.
3. ScmClient#getContainerLocations is changed to `getContainerLocations(Iterable<Long> containerIds, boolean forceRefresh, boolean sortDatanodes)`. getKeyInfo API now uses this and never calls sortDatanodes separately.

Issue Links

• relates to: HDDS-9343 Shift sortDatanodes logic to OM
• links to: GitHub Pull Request #5322