[CLOUDSTACK-5141] [Automation] Router deployment failed due to failure in SavePasswordCommand, observed error "Unable to save password to DomR" error in KVM agent log

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Cannot Reproduce
• Affects Version/s: 4.2.1
• Fix Version/s: 4.3.0
• Component/s: Virtual Router
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: KVM ; Branch : 4.2.1
• Assignee: Sheng Yang
• Reporter: Rayees Namathponnan
• Created: 12/Nov/13 04:24
• Updated: 31/Jan/14 18:50
• Resolved: 06/Dec/13 05:41

Description

router deployment faling inconsistently during automation run,

Command com.cloud.agent.api.routing.SavePasswordCommand failed in KVM agent

KVM agent log

2013-11-11 11:43:36,783 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Request:Seq 2-674103501: { Cmd , MgmtId: 29066118877352, via: 2, Ver: v
1, Flags: 100111, [{"com.cloud.agent.api.routing.SavePasswordCommand":{"password":"fnirq_cnffjbeq","vmIpAddress":"10.1.1.17","vmName":"494fbcec-82ab-489
3-95dc-36cb41f8c0ff","executeInSequence":true,"accessDetails":

{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-18-QA","rout er.ip":"169.254.1.19"}

,"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.17","vmName":"494fbcec-82ab-4893-95dc-36cb41f8c0ff
","executeInSequence":true,"accessDetails":

{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-18-QA","router.ip":"169.254.1.1 9"}

,"wait":0}}] }

2013-11-11 11:43:57,799 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) Seq 2-674103501: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":false,"details":"Unable to save password to DomR.","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] }

Management server log

2013-11-11 11:43:56,991 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-1:null) SeqA 6-270: Processing Seq 6-270: { Cmd , MgmtId: -1, via: 6, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":3,"_loadInfo":"

{\n \"connections\": []\n}

","wait":0}}] }
2013-11-11 11:43:56,994 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-1:null) SeqA 6-270: Sending Seq 6-270: { Ans: , MgmtId: 29066118877352, via: 6, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
2013-11-11 11:43:57,864 DEBUG [agent.transport.Request] (AgentManager-Handler-13:null) Seq 2-674103501: Processing: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 110, [{"com.cloud.agent.api.Answer":{"result":false,"details":"Unable to save password to DomR.","wait":0}},{"com.cloud.agent.api.Answer":{"result":false,"details":"Stopped by previous failure","wait":0}}] }
2013-11-11 11:43:57,864 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-13:null) Seq 2-674103504: Sending now. is current sequence.
2013-11-11 11:43:57,865 DEBUG [agent.transport.Request] (Job-Executor-98:job-90 = [ e7f1a0ac-f28a-4d3c-b5bf-8da87e6db622 ]) Seq 2-674103501: Received: { Ans: , MgmtId: 29066118877352, via: 2, Ver: v1, Flags: 110,

{ Answer, Answer }

}
2013-11-11 11:43:57,865 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-98:job-90 = [ e7f1a0ac-f28a-4d3c-b5bf-8da87e6db622 ]) Unable to contact resource.
com.cloud.exception.ResourceUnavailableException: Resource [DataCenter:1] is unreachable: Unable to apply userdata and password entry on router
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyRules(VirtualNetworkApplianceManagerImpl.java:3827)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.applyUserData(VirtualNetworkApplianceManagerImpl.java:3017)
at com.cloud.network.element.VirtualRouterElement.addPasswordAndUserdata(VirtualRouterElement.java:930)
at com.cloud.network.NetworkManagerImpl.prepareElement(NetworkManagerImpl.java:2085)
at com.cloud.network.NetworkManagerImpl.prepareNic(NetworkManagerImpl.java:2200)
at com.cloud.network.NetworkManagerImpl.prepare(NetworkManagerImpl.java:2136)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:886)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:227)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:209)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3440)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3000)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:2986)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:420)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)