[YUNIKORN-2031] metric yunikorn_root_.*_resource{state="pending"} is inaccurate

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.2.0, 1.3.0
• Fix Version/s: 1.4.0
• Component/s: core - scheduler
• Labels: pull-request-available
• Target Version: 1.4.0
• Assignee: Wei Huang
• Reporter: Wei Huang
• Created: 11/Oct/23 21:30
• Updated: 20/Nov/23 16:42
• Resolved: 24/Oct/23 22:58

Description

Given a sample static queue configmap yaml:

 

apiVersion: v1
data:
  queues.yaml: |
    partitions:
    - name: default
      queues:
      - name: root
        submitacl: '*'
        queues:
        - name: dev
          resources:
            max:
              memory: 20Gi
              vcore: "5"
        - name: sre
          resources:
            max:
              memory: 20Gi
              vcore: "5"
kind: ConfigMap
metadata:
  name: yunikorn-configs
  namespace: default 

 

and a Pod template yaml like:

 

kind: Pod
apiVersion: v1
metadata:
  generateName: dev-
  labels:
    applicationId: dev
    queue: root.dev
spec:
  schedulerName: yunikorn
  containers:
  - name: pause
    image: registry.k8s.io/pause
    resources:
      requests:
        cpu: 1
      limits:
        cpu: 1 

 

If I create 6 pods:

 

for i in {1..6}; do k create -f pods/dev-pod.yaml; done 

5 Pods are in Running state, 1 is in Pending state. Which looks good as dev queue's max quota is 5 cpus.

 

However, if I check the metric yunikorn_root_dev_queue_resource, it shows incorrect pending pods/resource:

Metric	Value
yunikorn_root_dev_queue_resource {instance="localhost:9080", job="yunikorn", resource="pods", state="allocated"}	5
yunikorn_root_dev_queue_resource {instance="localhost:9080", job="yunikorn", resource="pods", state="pending"}	2
yunikorn_root_dev_queue_resource {instance="localhost:9080", job="yunikorn", resource="vcore", state="allocated"}	5000
yunikorn_root_dev_queue_resource {instance="localhost:9080", job="yunikorn", resource="vcore", state="pending"}	2000

Issue Links

• causes: YUNIKORN-2071 Remove updateAllocatedAndPendingResourceMetrics()
• links to: GitHub Pull Request #673