[YARN-7800] Bind node constraint once a container is proposed to be placed on this node

• Type: Sub-task
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: RM
• Assignee: Weiwei Yang
• Reporter: Weiwei Yang
• Created: 24/Jan/18 03:18
• Updated: 03/Feb/19 03:23

Description

We found when there is circular dependency between multiple scheduling requests, allocation decisions made by placement constraint algorithm might be conflicting with related tags. More describing of the issue please refer to YARN-7783.

To solve this issue, a possible solution is to bind node constraint. If the algorithm wants to place any new container to this node, except for checking if it satisfies the placement constraint, it also check if it satisfies the node constraint. For example

1) "foo", anti-affinity with "foo"

Implies node constraint: on each node, it cannot have more than 1 foo tags

2) "bar", anti-affinity with "foo"

Implies node constraint: on each node, it cannot have both "bar" and "foo" tags

With such constraint, it works like

• req2 is placed on any of nodes, e.g n2, a node constraint[1] is added to n2 that constrains this node cannot have both "bar" and "foo" tags
• when the algorithm wants to place req1 on n2, it checks if its placement constraint is satisfied. It should be as there is no foo container on this node yet.
• Then the algorithm checks if the node constraint is satisfied. It is not because it violate node constraint [1].

This avoids to do additional re-attempt like what was done in YARN-7783.

Issue Links

• is related to: YARN-6621 Validate Placement Constraints