[CLOUDSTACK-3538] [Automation]Router and SSVM not coming up after restart - ASF JIRA

Details

Type: Bug
Status: Closed
Priority: Critical
Resolution: Fixed
Affects Version/s: 4.2.0
Fix Version/s: 4.2.0
Component/s: None
Security Level: Public (Anyone can view this level - this is the default.)
Labels:
None
Environment:
KVM
Build 4.2

Description

Below BVT test cases failed in latest 4.2 run

integration.smoke.test_network.TestRebootRouter.test_reboot_router
integration.smoke.test_network.TestRebootRouter.test_reboot_router
integration.smoke.test_routers.TestRouterServices.test_08_start_router
integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm

All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,

Please see the attached log after restart router "r-133-QA"
-----------------------------------------------------------------------------------

1) Issued stop command @ 11:42:02

2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Execution is successful.
2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Try to stop the vm at first
2013-07-15 11:42:10,777 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) successfully shut down vm r-133-QA
2013-07-15 11:42:52,454 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Ignoring VM r-133-QA in transition state stopping.
2013-07-15 11:42:52,455 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
2013-07-15 11:42:52,759 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Execution is successful.

2) Started Command issue after 15 minute @ 11:58:03

2013-07-15 11:58:03,238 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 169.254.0.187 -c
2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 5-1532821715: { Cmd , MgmtId: 29066118877352, via: 5, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":133,"name":"r-133-QA","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-133-QA eth2ip=10.223.122.92 eth2mask=255.255.255.192 gateway=10.223.122.65 eth0ip=10.1.1.1 eth0mask=255.255.255.0 domain=cs43auto.advanced dhcprange=10.1.1.1 eth1ip=169.254.1.214 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"ee61a6681d35c765","params":{},"uuid":"240b6249-e20f-4c35-93b4-024303bef80a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c91d846-6310-4440-aa1d-67b35f2d2c36","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"fff90cb5-06dd-33b3-8815-d78c08ca01d9","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/rayees/SC_QA_AUTO4/primary","port":2049}},"name":"ROOT-133","size":147456,"path":"d4476933-b42c-45f4-a061-22c1908edca5","volumeId":138,"vmName":"r-133-QA","accountId":67,"format":"QCOW2","id":138}},"diskSeq":0,"type":"ROOT"}],"nics":[

{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"1b55ad5b-8516-4e79-90a4-33052c43b0e0","ip":"10.223.122.92","netmask":"255.255.255.192","gateway":"10.223.122.65","mac":"06:a0:90:00:00:55","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false}

,

{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"859700c2-124b-4183-a64c-8357f57a7671","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:21:a1:00:02","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2311","isolationUri":"vlan://2311","isSecurityGroupEnabled":false}

,

{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"4ce6c02e-6df6-4960-857c-3e5fdd2d8e1b","ip":"169.254.1.214","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:d6","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}

]},"hostIp":"10.223.50.67","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.1.214","port":3922,"interval":6,"retries":100,"name":"r-133-QA","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":

{"router.name":"r-133-QA","router.ip":"169.254.1.214"}

,"wait":0}},{},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[

{"accountId":67,"publicIp":"10.223.122.92","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1221","vlanGateway":"10.223.122.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:92:84:00:00:55","networkRate":200,"trafficType":"Public"}

],"accessDetails":

{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"}

,"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:39:b4:00:01","vmIpAddress":"10.1.1.120","vmName":"dVM","defaultRouter":"10.1.1.1","defaultDns":"10.1.1.1","duid":"00:03:00:01:02:00:39:b4:00:01","isDefault":true,"executeInSequence":false,"accessDetails":

{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.1.214","router.name":"r-133-QA"}

,"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.120","vmName":"dVM","executeInSequence":false,"accessDetails":

{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"}

,"wait":0}}] }
2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.StartCommand
2013-07-15 11:58:03,363 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.

3) r-133-QA still in starting state in UI , even after 20 minute

See the "virsh list" from the host, r-133-QA is missing

[root@Rack2Host12 agent]# virsh list
Id Name State
----------------------------------------------------
3 r-130-QA running
5 r-128-QA running
6 r-126-QA running
7 i-65-129-QA running
10 i-71-139-QA running
11 r-57-QA running
13 r-144-QA running

Also below refer below check-in this might be caused this
https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c

[Automation]Router and SSVM not coming up after restart

Details

Description

Attachments

Attachments

Activity

People

Dates