[YUNIKORN-2156] Improvement for gang scheduling when spark dynamic allocation enabled

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: core - scheduler
• Target Version: 1.7.0
• Assignee: Unassigned
• Reporter: Qi Zhu
• Created: 15/Nov/23 08:37
• Updated: 09/Sep/24 11:47

Description

Try to improve the case, Spark Dynamic Allocation when we enable gang scheduling:
If we can add some improvements to this case? Such as:
For preemption case:

lower priority for anything more than the gang and thus get them killed earlier than when they do not belong to a gang

For quota hit:

For normal gang scheduling, we will reject jobs with total gang size > quota, but for dynamic allocations, if we can add monitor logic to monitor the exceeded resource allocations for those jobs?

• We may extend the quota some value when it happens?