[STRATOS-919] Error while AS asks CC to terminate an instance that are not in the topology

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.1.0 M2
• Fix Version/s: 4.1.0 RC3
• Component/s: Autoscaler, Cloud Controller, Messaging
• Assignee: Unassigned
• Reporter: Rajkumar Rajaratnam
• Created: 28/Oct/14 11:28
• Updated: 31/May/15 07:14
• Resolved: 10/Mar/15 02:35

Description

The following scenario will ended up with termination failed error continuously.

AS asks CC to create a container
CC schedule a new task and returning member contexts to AS
AS adding the members to pending list
CC is also adding member contexts to its data holder
But CC will not add it to the topology until pods status become to "Running" state
Meanwhile members are exceeding their timeouts and moved to obsolete list
AS asks CC to terminate obsolete pods
CC kills it, but didn't send the member terminated event, because member is not in the topology
AS didn't get any member terminated event, so it will not remove any members from its member lists
AS will ask CC again to terminate the pod, in next monitor interval
CC now will complain "Failed to terminate member. Member id not found", because it terminated those pods already and removed the member contexts from its data holder
So this "Failed to terminate member. Member id not found" will occur continuously