Uploaded image for project: 'Apache YuniKorn'
  1. Apache YuniKorn
  2. YUNIKORN-2327

Race condition during update Occupied Resource from Shim to Core

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Delivered
    • None
    • None
    • None

    Description

      When initializing YuniKorn, existing Non-YuniKorn pods (ForeignPods) are counted as node's occupied resources. An SchedulerAPI.UpdateNode(request) is triggered asynchronously to update the occupied resources for the node in the core. However, a race condition occurs on the core side during this asynchronous update process.

      How to reproduce:

      • Add 2 seconds delay for the first pod, the final occupied resource will equal to the first pod's resource size after restart YuniKorn. (example)
      • The issue is the root cause of YUNIKORN-2313

      Error Logs: (E2E test link-v1.29.0)
      Shim logs: (yk8s-worker)

      2024-01-15T14:39:20.135Z Shim trigger SchedulerAPI() request: occupied: resources:{key:"pods" value:{value:1}}
      2024-01-15T14:39:20.135Z Shim trigger SchedulerAPI() request: occupied: resources:{key:"memory" value:{value:52428800}} resources:{key:"pods" value:{value:2}} resources:{key:"vcore" value:{value:100}}
      2024-01-15T14:39:20.136Z Shim trigger SchedulerAPI() request: occupied: resources:{key:"memory" value:{value:576716800}} resources:{key:"pods" value:{value:3}} resources:{key:"vcore" value:{value:200}}
      

      Core logs: (yk8s-worker)

      2024-01-15T14:39:20.137Z set occupiedResource: map[memory:52428800 pods:2 vcore:100]
      2024-01-15T14:39:20.137Z set occupiedResource: map[memory:576716800 pods:3 vcore:200]
      2024-01-15T14:39:22.136Z set occupiedResource: map[pods:1]
      

      Final occupied resource in state dump:

      ...
              {
                "nodeID": "yk8s-worker",
                "attributes": {
                  "ready": "true",
                  "si.io/hostname": "yk8s-worker",
                  "si.io/rackname": "/rack-default",
                  "si/node-partition": "[mycluster]default"
                },
                 ...
                "occupied": {
                  "pods": 1
                }...
              }
      

      Key code of this issue:

      Attachments

        Issue Links

          Activity

            People

              Yu-Lin Chen Yu-Lin Chen
              Yu-Lin Chen Yu-Lin Chen
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: