[YUNIKORN-646] Add metrics implementation: "allocating_latency_seconds"

• Type: Sub-task
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 0.11
• Component/s: core - common
• Labels: pull-request-available
• Target Version: 0.11
• Assignee: Chenya Zhang
• Reporter: Chenya Zhang
• Created: 21/Apr/21 00:18
• Updated: 21/Jan/22 21:48
• Resolved: 11/May/21 20:46

Description

Observation:

1. Container allocating latency stays at 0. The number of allocation attempts fluctuates normally.
2. Scheduler metrics definition is not consistent and sometimes hard to understand.

Root cause analysis:

1. The metrics "allocating_latency_seconds" is not fully implemented or the implementation is missed in recent releases. For example, ObserveSchedulingLatency() is currently not called when allocating containers.
2. Scheduler metrics is implemented by multiple developers in the past while not following the same convention.

Improvement Plan:

1. The top level container allocation latency can be captured by the main scheduling routine in scheduler/context.go. Reason: The schedule() method in scheduler/context.go is the entry point to process each partition in the scheduler, walk over each queue and app to check if anything can be scheduled.
2. The metrics name "allocating_latency_seconds" can be changed to "scheduling_latency_seconds". Reason: The metrics is initially defined as "schedulingLatency" in metrics/scheduler.go. Naming consistency can help to avoid confusion.
3. Other metrics definition and help message can be improved to make metrics/scheduler.go consistent. (Open to create a separate PR for the refactoring work.)
4. New metrics can be further added to monitor lower level latency when the scheduler is iterating over partition list, queues, applications, requests etc. Not included in this PR.

Issue Links

• links to: GitHub Pull Request #273