[YUNIKORN-1105] Rethink memory resource conversion to MB

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Assignee: Craig Condit
• Reporter: Wilfred Spiegelenburg
• Created: 04/Mar/22 06:03
• Updated: 04/May/22 23:52
• Resolved: 17/Mar/22 07:01

Description

The choice to represent memory in units of MB and not in bytes comes with a side effect. We convert a pod or node memory size into something that is a MB (10 based).

Not everything is expressible in whole MB. We round up to the nearest MB. 1 byte over and we use the whole MB. This means that a node looks larger in YuniKorn than it really is. It is less than a MB but it can still cause an issue with a pod just fitting or not fitting.

As an example: a pods needs exactly 10MB, 10,000,000 bytes. A node in YuniKorn shows 10MB free but in reality it is not 10,000,000 bytes but only 9,000,001.
It can also happen the other way around. The pod asks for 9,000,001 bytes, YuniKorn sees it as 10MB. The node in YuniKorn shows 9MB free but in reality the node has 9,500,000 free as a previous pod we have scheduled did not use 10MB but only 9,500,000. YuniKorn fails to place the pod, The auto scaler says there is enough room to place the pod.
I know I am splitting hairs here but it is a real possibility. These failures are really hard to track down and link back. YuniKorn schedules the pod and the node bind fails with not enough resources or scale up fails to trigger when expected.

With the choice of milli for cpu we have far less of an issue as K8s does not support more than 3 decimal places. In other words the smallest value used in K8s is 1m for cpu.

Issue Links

• is a parent of: YUNIKORN-1131 Document behaviour of resource in REST API
• relates to: YUNIKORN-165 API response fields can be returned as proper data structure rather than free text
• links to: GitHub Pull Request #71
• links to: GitHub Pull Request #386