CloudStack
  1. CloudStack
  2. CLOUDSTACK-3538

[Automation]Router and SSVM not coming up after restart

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 4.2.0
    • Fix Version/s: 4.2.0
    • Component/s: None
    • Security Level: Public (Anyone can view this level - this is the default.)
    • Labels:
      None
    • Environment:
      KVM
      Build 4.2

      Description

      Below BVT test cases failed in latest 4.2 run

      integration.smoke.test_network.TestRebootRouter.test_reboot_router
      integration.smoke.test_network.TestRebootRouter.test_reboot_router
      integration.smoke.test_routers.TestRouterServices.test_08_start_router
      integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
      integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm

      All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,

      Please see the attached log after restart router "r-133-QA"
      -----------------------------------------------------------------------------------

      1) Issued stop command @ 11:42:02

      2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Execution is successful.
      2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Try to stop the vm at first
      2013-07-15 11:42:10,777 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) successfully shut down vm r-133-QA
      2013-07-15 11:42:52,454 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Ignoring VM r-133-QA in transition state stopping.
      2013-07-15 11:42:52,455 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
      2013-07-15 11:42:52,759 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Execution is successful.

      2) Started Command issue after 15 minute @ 11:58:03

      2013-07-15 11:58:03,238 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 169.254.0.187 -c
      2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 5-1532821715: { Cmd , MgmtId: 29066118877352, via: 5, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":133,"name":"r-133-QA","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-133-QA eth2ip=10.223.122.92 eth2mask=255.255.255.192 gateway=10.223.122.65 eth0ip=10.1.1.1 eth0mask=255.255.255.0 domain=cs43auto.advanced dhcprange=10.1.1.1 eth1ip=169.254.1.214 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"ee61a6681d35c765","params":{},"uuid":"240b6249-e20f-4c35-93b4-024303bef80a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c91d846-6310-4440-aa1d-67b35f2d2c36","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"fff90cb5-06dd-33b3-8815-d78c08ca01d9","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/rayees/SC_QA_AUTO4/primary","port":2049}},"name":"ROOT-133","size":147456,"path":"d4476933-b42c-45f4-a061-22c1908edca5","volumeId":138,"vmName":"r-133-QA","accountId":67,"format":"QCOW2","id":138}},"diskSeq":0,"type":"ROOT"}],"nics":[

      {"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"1b55ad5b-8516-4e79-90a4-33052c43b0e0","ip":"10.223.122.92","netmask":"255.255.255.192","gateway":"10.223.122.65","mac":"06:a0:90:00:00:55","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false}

      ,

      {"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"859700c2-124b-4183-a64c-8357f57a7671","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:21:a1:00:02","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2311","isolationUri":"vlan://2311","isSecurityGroupEnabled":false}

      ,

      {"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"4ce6c02e-6df6-4960-857c-3e5fdd2d8e1b","ip":"169.254.1.214","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:d6","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}

      ]},"hostIp":"10.223.50.67","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.1.214","port":3922,"interval":6,"retries":100,"name":"r-133-QA","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":

      {"router.name":"r-133-QA","router.ip":"169.254.1.214"}

      ,"wait":0}},{},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[

      {"accountId":67,"publicIp":"10.223.122.92","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1221","vlanGateway":"10.223.122.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:92:84:00:00:55","networkRate":200,"trafficType":"Public"}

      ],"accessDetails":

      {"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"}

      ,"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:39:b4:00:01","vmIpAddress":"10.1.1.120","vmName":"dVM","defaultRouter":"10.1.1.1","defaultDns":"10.1.1.1","duid":"00:03:00:01:02:00:39:b4:00:01","isDefault":true,"executeInSequence":false,"accessDetails":

      {"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.1.214","router.name":"r-133-QA"}

      ,"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.120","vmName":"dVM","executeInSequence":false,"accessDetails":

      {"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"}

      ,"wait":0}}] }
      2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.StartCommand
      2013-07-15 11:58:03,363 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.

      3) r-133-QA still in starting state in UI , even after 20 minute

      See the "virsh list" from the host, r-133-QA is missing

      [root@Rack2Host12 agent]# virsh list
      Id Name State
      ----------------------------------------------------
      3 r-130-QA running
      5 r-128-QA running
      6 r-126-QA running
      7 i-65-129-QA running
      10 i-71-139-QA running
      11 r-57-QA running
      13 r-144-QA running

      Also below refer below check-in this might be caused this
      https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c

      1. CLOUDSTACK-3538.rar
        3.61 MB
        Rayees Namathponnan
      2. CLOUDSTACK-3538_Debug.rar
        175 kB
        Rayees Namathponnan

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            edison su
            Reporter:
            Rayees Namathponnan
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development