[YUNIKORN-876] Optimize scheduler memory consumption

• Type: Improvement
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: core - scheduler, shim - kubernetes
• Labels: performance pull-request-available
• Target Version: 1.0.0
• Assignee: Craig Condit
• Reporter: Weiwei Yang
• Created: 13/Oct/21 17:06
• Updated: 04/May/22 23:52
• Resolved: 15/Feb/22 20:50

Description

On a cluster with about 500 nodes, we've seen the memory consumption goes over 12GB. This is not optimal. Using http://yunikorn.apache.org/docs/next/performance/profiling#memory-profiling, we got the memory footprint dump as attached in the JIRA. We need to optimize resource consumption and reduce the memory usage.

Issue Links

• is a parent of: YUNIKORN-1100 Fix inconsistencies and leaks in shim caches
• is duplicated by: YUNIKORN-1070 Potential scheduler memory leak
• links to: GitHub Pull Request #365