Uploaded image for project: 'Apache YuniKorn'
  1. Apache YuniKorn
  2. YUNIKORN-2977 [Umbrella] DaemonSet preemption hardening
  3. YUNIKORN-2980

DaemonSet preemption: don't flood the logs if victim selection fails

    XMLWordPrintableJSON

Details

    Description

      If we can't find a proper victim for a DaemonSet pod, we constaly print the following to the console in every scheduling cycle:

      log.Log(log.SchedApplication).Info("Triggering preemption process for daemon set ask",
      		zap.String("ds allocation key", ask.GetAllocationKey()))
      [...]
      log.Log(log.SchedApplication).Warn("Problem in finding the victims for preempting resources to meet required ask requirements",
      		zap.String("ds allocation key", ask.GetAllocationKey()),
      		zap.String("node id", reserve.nodeID))
      

      Since we attempt to schedule in every 100ms, this is logged at least 10 times in every second.

      Suggestion: do the preemption silently and only log if it has succeeded, just like regular preemption. Otherwise, use Allocation.LogAllocationFailure() which we already do for a variety of things.

      Attachments

        Issue Links

          Activity

            People

              pbacsko Peter Bacsko
              pbacsko Peter Bacsko
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: