[CLOUDSTACK-6309] [Automation] Router deployment failed due to "aggregated command s execution failure"

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 4.4.0
• Fix Version/s: 4.4.0, 4.5.0
• Component/s: Network Controller
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: KVM 6.3; Build 4.4
• Assignee: Sheng Yang
• Reporter: Rayees Namathponnan
• Created: 31/Mar/14 15:20
• Updated: 21/Sep/14 19:43
• Resolved: 31/Mar/14 18:30

Description

This issues in observed with automation run, many router deployment failures are observed with error

2014-03-31 00:16:52,144 WARN [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Failed to re-
program the network as a part of network Ntwk[38591d1e-5193-451c-beb0-854f44692880|Guest|8] implement due to aggregated command
s execution failure!

Steps to reproduce

1) Create an account
2) Deploy new Newwork and VM

Result

Router deployment failed with below error in MS

2014-03-31 00:16:51,661 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Seq 1-6623669151955157207:
Sending { Cmd , MgmtId: 29066118877352, via: 1(Rack2Host11.lab.vmops.com), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.rout
ing.AggregationControlCommand":{"action":"Finish","accessDetails":

{"router.guest.ip":"10.1.1.1","router.name":"r-21-QA","router .ip":"169.254.3.129"}

,"wait":0}}] }

2014-03-31 00:16:52,144 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Seq 1-6623669151955157207: Received: { Ans: , MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 10,

{ Answer } }
2014-03-31 00:16:52,144 WARN [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Failed to re-program the network as a part of network Ntwk[38591d1e-5193-451c-beb0-854f44692880|Guest|8] implement due to aggregated commands execution failure!
2014-03-31 00:16:52,146 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Service SecurityGroup is not supported in the network id=217
2014-03-31 00:16:52,158 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Seq 1-6623669151955157209: Sending { Cmd , MgmtId: 29066118877352, via: 1(Rack2Host11.lab.vmops.com), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Cleanup","accessDetails":{"router.guest.ip":"10.1.1.1","router.name":"r-21-QA","router.ip":"169.254.3.129"},"wait":0}}] }
2014-03-31 00:16:52,202 DEBUG [c.c.a.t.Request] (AgentManager-Handler-9:null) Seq 1-6623669151955157209: Processing: { Ans: , MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }
2014-03-31 00:16:52,202 DEBUG [c.c.a.t.Request] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Seq 1-6623669151955157209: Received: { Ans: , MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 10, { Answer }

}
2014-03-31 00:16:52,202 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Cleaning up because we're unable to implement the network Ntwk[38591d1e-5193-451c-beb0-854f44692880|Guest|8]
2014-03-31 00:16:52,209 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Lock is acquired for network Ntwk[38591d1e-5193-451c-beb0-854f44692880|Guest|8] as a part of network shutdown
2014-03-31 00:16:52,227 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Releasing 0 port forwarding rules for network id=217 as a part of shutdownNetworkRules
2014-03-31 00:16:52,227 DEBUG [c.c.n.f.FirewallManagerImpl] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) There are no rules to forward to the network elements
2014-03-31 00:16:52,228 DEBUG [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-22:Job-105/Job-106 ctx-1996e983) Releasing 0 static nat rules for network id=217 as a part of shutdownNetworkRules

Below is is the answer from agent (6623669151955157207)

2014-03-30 23:17:41,320 WARN [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) Timed out: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh vr_cfg.sh 169.254.3.129 -c /var/cache/cloud/VR-7f685597-fefe-407f-a767-d0e841f597a3.cfg . Output is:
2014-03-30 23:17:41,320 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Seq 1-6623669151955157207: { Ans: , MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":false,"details":"process hasn't exited","wait":0}}] }
2014-03-30 23:17:41,339 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 1-6623669151955157209: { Cmd , MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.routing.AggregationControlCommand":{"action":"Cleanup","accessDetails":

{"router.guest.ip":"10.1.1.1","router.ip":"169.254.3.129","router.name":"r-21-QA"}

,"wait":0}}] }
2014-03-30 23:17:41,339 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.routing.AggregationControlCommand
2014-03-30 23:17:41,339 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Seq 1-6623669151955157209: { Ans: , MgmtId: 29066118877352, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] }