[YUNIKORN-704] [Umbrella] Use the same mechanism to schedule daemon set pods as the default scheduler

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 0.12.1
• Component/s: shim - kubernetes
• Target Version: 0.12.1
• Assignee: TingYao Huang
• Reporter: Chaoran Yu
• Created: 12/Jun/21 17:23
• Updated: 05/Sep/23 16:35
• Resolved: 26/Aug/21 07:00

Description

We sometimes see DaemonSet pods fail to be scheduled. Please see attached files for the YAML and kubectl describe output of one such pod. We originally suspected node reservation was to blame. But even after setting the DISABLE_RESERVATION environment variable to true, we still see such scheduling failures. The issue is especially severe when K8s nodes have disk pressure that causes lots of pods to be evicted. Newly created pods will stay in pending forever. We have to temporarily uninstall YuniKorn and let the default scheduler do the scheduling for these pods. 

This issue is critical because lots of important pods belong to a DaemonSet, such as Fluent Bit, a common logging solution. This is probably the last remaining roadblock for us to have the confidence to have YuniKorn entirely replace the default scheduler.

Issue Links

• is related to: YUNIKORN-1085 DaemonSet pods may fail to be scheduled on new nodes added during autoscaling

Sub-Tasks

• Discover daemonset pods and tag with ignoreUnschedulableNodes attributes: Closed
• Ignore unschedulable nodes for asks with certain tags: Closed
• Add a field in scheduler-interface to represent ignoring unschedulable nodes for the scheduler: Closed
• Add a field in scheduler-interface to represent required node for the scheduler: Closed
• Discover daemonset pods and parse nodeaffinity value to allocationAsk tags: Closed
• Handle required node tag for asks with certain tags: Closed
• add `KeyIgnoreUnschedulable` to scheduler-interface-spec.md: Closed