[YUNIKORN-2933] Don't add duplicated taskGroup to app

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Not A Bug
• Affects Version/s: 1.3.0, 1.5.2
• Component/s: shim - kubernetes
• Labels: pull-request-available
• Assignee: Unassigned
• Reporter: Xiaobao Wu
• Created: 17/Oct/24 06:49
• Updated: 29/Oct/24 06:41
• Resolved: 29/Oct/24 06:18

Description

    When the app processes the yunikorn.apache.org/task-groups annotation on Pod, it did not consider the case of taskgroup with the same name (i.e. taskgroup with the same name).For example, the following task-groups information is defined:

 

yunikorn.apache.org/task-groups='
[{
        "name": "spark-executor",
        "minMember": 2,
        "minResource": {
            "cpu": "1",
            "memory": "1Gi"
        }
    }, {
        "name": "spark-driver",
        "minMember": 1,
        "minResource": {
            "cpu": "1",
            "memory": "1Gi"
        }
    },
    {
        "name": "spark-executor",
        "minMember": 1,
        "minResource": {
            "cpu": "2",
            "memory": "2Gi"
        }
    }
]
'

    From the above example, it can be seen that there are two task-group with the same name in the task-groups. So, after these ph Tasks are created, if an executor task (2C 2G)
try to allocate placeholder‘s resource, it may erroneously release the ph task in the spark-executor group with a resource specification of 1C 1G (because the resource specification cannot meet 2C 2G). In this regard, I think that before properly handling ph tasks with different resource specifications within the same task group, it is necessary to avoid having two task-group with the same name but different resource specifications in the app.
 

Issue Links

• relates to: YUNIKORN-2517 [Yunikorn] Incorrect Placeholder Count for Duplicate Task Groups in Gang scheduling
• links to: GitHub Pull Request #928