CloudStack
  1. CloudStack
  2. CLOUDSTACK-3538

[Automation]Router and SSVM not coming up after restart

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 4.2.0
    • Fix Version/s: 4.2.0
    • Component/s: None
    • Security Level: Public (Anyone can view this level - this is the default.)
    • Labels:
      None
    • Environment:
      KVM
      Build 4.2

      Description

      Below BVT test cases failed in latest 4.2 run

      integration.smoke.test_network.TestRebootRouter.test_reboot_router
      integration.smoke.test_network.TestRebootRouter.test_reboot_router
      integration.smoke.test_routers.TestRouterServices.test_08_start_router
      integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
      integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm

      All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,

      Please see the attached log after restart router "r-133-QA"
      -----------------------------------------------------------------------------------

      1) Issued stop command @ 11:42:02

      2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Execution is successful.
      2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Try to stop the vm at first
      2013-07-15 11:42:10,777 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) successfully shut down vm r-133-QA
      2013-07-15 11:42:52,454 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Ignoring VM r-133-QA in transition state stopping.
      2013-07-15 11:42:52,455 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
      2013-07-15 11:42:52,759 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Execution is successful.

      2) Started Command issue after 15 minute @ 11:58:03

      2013-07-15 11:58:03,238 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 169.254.0.187 -c
      2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 5-1532821715: { Cmd , MgmtId: 29066118877352, via: 5, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":133,"name":"r-133-QA","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-133-QA eth2ip=10.223.122.92 eth2mask=255.255.255.192 gateway=10.223.122.65 eth0ip=10.1.1.1 eth0mask=255.255.255.0 domain=cs43auto.advanced dhcprange=10.1.1.1 eth1ip=169.254.1.214 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"ee61a6681d35c765","params":{},"uuid":"240b6249-e20f-4c35-93b4-024303bef80a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c91d846-6310-4440-aa1d-67b35f2d2c36","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"fff90cb5-06dd-33b3-8815-d78c08ca01d9","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/rayees/SC_QA_AUTO4/primary","port":2049}},"name":"ROOT-133","size":147456,"path":"d4476933-b42c-45f4-a061-22c1908edca5","volumeId":138,"vmName":"r-133-QA","accountId":67,"format":"QCOW2","id":138}},"diskSeq":0,"type":"ROOT"}],"nics":[

      {"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"1b55ad5b-8516-4e79-90a4-33052c43b0e0","ip":"10.223.122.92","netmask":"255.255.255.192","gateway":"10.223.122.65","mac":"06:a0:90:00:00:55","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false}

      ,

      {"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"859700c2-124b-4183-a64c-8357f57a7671","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:21:a1:00:02","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2311","isolationUri":"vlan://2311","isSecurityGroupEnabled":false}

      ,

      {"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"4ce6c02e-6df6-4960-857c-3e5fdd2d8e1b","ip":"169.254.1.214","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:d6","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}

      ]},"hostIp":"10.223.50.67","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.1.214","port":3922,"interval":6,"retries":100,"name":"r-133-QA","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":

      {"router.name":"r-133-QA","router.ip":"169.254.1.214"}

      ,"wait":0}},{},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[

      {"accountId":67,"publicIp":"10.223.122.92","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1221","vlanGateway":"10.223.122.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:92:84:00:00:55","networkRate":200,"trafficType":"Public"}

      ],"accessDetails":

      {"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"}

      ,"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:39:b4:00:01","vmIpAddress":"10.1.1.120","vmName":"dVM","defaultRouter":"10.1.1.1","defaultDns":"10.1.1.1","duid":"00:03:00:01:02:00:39:b4:00:01","isDefault":true,"executeInSequence":false,"accessDetails":

      {"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.1.214","router.name":"r-133-QA"}

      ,"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.120","vmName":"dVM","executeInSequence":false,"accessDetails":

      {"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"}

      ,"wait":0}}] }
      2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.StartCommand
      2013-07-15 11:58:03,363 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.

      3) r-133-QA still in starting state in UI , even after 20 minute

      See the "virsh list" from the host, r-133-QA is missing

      [root@Rack2Host12 agent]# virsh list
      Id Name State
      ----------------------------------------------------
      3 r-130-QA running
      5 r-128-QA running
      6 r-126-QA running
      7 i-65-129-QA running
      10 i-71-139-QA running
      11 r-57-QA running
      13 r-144-QA running

      Also below refer below check-in this might be caused this
      https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c

      1. CLOUDSTACK-3538.rar
        3.61 MB
        Rayees Namathponnan
      2. CLOUDSTACK-3538_Debug.rar
        175 kB
        Rayees Namathponnan

        Activity

        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        2d 1h 37m 1 Animesh Chaturvedi 17/Jul/13 21:33
        Resolved Resolved Reopened Reopened
        1d 18h 11m 1 Rayees Namathponnan 19/Jul/13 15:44
        Reopened Reopened Resolved Resolved
        20d 9h 37m 1 edison su 09/Aug/13 01:22
        Resolved Resolved Closed Closed
        23h 24m 1 Rayees Namathponnan 10/Aug/13 00:46
        Rayees Namathponnan made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        Rayees Namathponnan added a comment -

        Not found this issue in last autoamtion runs

        Show
        Rayees Namathponnan added a comment - Not found this issue in last autoamtion runs
        Hide
        Wei Zhou added a comment -

        Edison, it looks the passcmdline will succeed even in the first start of VM.
        I am not sure whether the issue CLOUDSTACK-2823 will appear again.
        Hope someone can have a test on CentOS 6.4 or befre.

        Show
        Wei Zhou added a comment - Edison, it looks the passcmdline will succeed even in the first start of VM. I am not sure whether the issue CLOUDSTACK-2823 will appear again. Hope someone can have a test on CentOS 6.4 or befre.
        Hide
        ASF subversion and git services added a comment -

        Commit ec2eafdfde1b70100391f56528b73f7a54bb4a7b in branch refs/heads/master from edison su
        [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=ec2eafd ]

        CLOUDSTACK-3538: if passcmdline succeed, don't need to retry again, and increase the retry to 5 minutes at most

        Show
        ASF subversion and git services added a comment - Commit ec2eafdfde1b70100391f56528b73f7a54bb4a7b in branch refs/heads/master from edison su [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=ec2eafd ] CLOUDSTACK-3538 : if passcmdline succeed, don't need to retry again, and increase the retry to 5 minutes at most
        edison su made changes -
        Status Reopened [ 4 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        ASF subversion and git services added a comment -

        Commit 2b32477acaf1715cb8e15cd601cf7404ab6e232d in branch refs/heads/4.2 from edison su
        [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=2b32477 ]

        CLOUDSTACK-3538: if passcmdline succeed, don't need to retry again, and increase the retry to 5 minutes at most

        Show
        ASF subversion and git services added a comment - Commit 2b32477acaf1715cb8e15cd601cf7404ab6e232d in branch refs/heads/4.2 from edison su [ https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;h=2b32477 ] CLOUDSTACK-3538 : if passcmdline succeed, don't need to retry again, and increase the retry to 5 minutes at most
        Animesh Chaturvedi made changes -
        Priority Major [ 3 ] Critical [ 2 ]
        Hide
        Animesh Chaturvedi added a comment -

        Seems like critical to me bumping up priority

        Show
        Animesh Chaturvedi added a comment - Seems like critical to me bumping up priority
        Rayees Namathponnan made changes -
        Resolution Fixed [ 1 ]
        Status Resolved [ 5 ] Reopened [ 4 ]
        Hide
        Rayees Namathponnan added a comment -

        Edison - reopening the defect as per the discussion, please close this defect after you make the changes

        Show
        Rayees Namathponnan added a comment - Edison - reopening the defect as per the discussion, please close this defect after you make the changes
        Rayees Namathponnan made changes -
        Priority Blocker [ 1 ] Major [ 3 ]
        Rayees Namathponnan made changes -
        Assignee edison su [ edison ]
        Hide
        Rayees Namathponnan added a comment - - edited

        Yes, i didnt find the issue with latest build

        But as per Edison, this is not a complete fix, we need to wait system vm to comeup before checking timeout, he will make the changes today.

        I will close this defect after this, removing blocker and assigning defect to edison

        Show
        Rayees Namathponnan added a comment - - edited Yes, i didnt find the issue with latest build But as per Edison, this is not a complete fix, we need to wait system vm to comeup before checking timeout, he will make the changes today. I will close this defect after this, removing blocker and assigning defect to edison
        Animesh Chaturvedi made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        Animesh Chaturvedi added a comment -

        Resolving as per Prasanna's comment. Rayees can you retry and reopen if needed

        Show
        Animesh Chaturvedi added a comment - Resolving as per Prasanna's comment. Rayees can you retry and reopen if needed
        Hide
        Prasanna Santhanam added a comment -

        Appears to be fixed in the latest run:
        http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-smoke-matrix/suite=test_ssvm/683/

        Previous KVM runs for test_ssvm just aborted.

        Show
        Prasanna Santhanam added a comment - Appears to be fixed in the latest run: http://jenkins.buildacloud.org/view/cloudstack-qa/job/test-smoke-matrix/suite=test_ssvm/683/ Previous KVM runs for test_ssvm just aborted.
        Hide
        Wei Zhou added a comment -

        Rayees,

        I notice Edison changed something in commit ba4c4400b5095e9c14e986f0cdfe6c7d1fd861a8, Could you test it again?

        [root@weizhou-centos master]# git show ba4c4400b5095e9c14e986f0cdfe6c7d1fd861a8 plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
        commit ba4c4400b5095e9c14e986f0cdfe6c7d1fd861a8
        Author: Edison Su <sudison@gmail.com>
        Date: Tue Jul 16 18:04:29 2013 -0700

        be able to upload template into swift

        diff --git a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
        index e51fbda..da86612 100755
        — a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
        +++ b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java
        @@ -1072,13 +1072,13 @@ ServerResource {

        private void passCmdLine(String vmName, String cmdLine)
        throws InternalErrorException {

        • final Script command = new Script(_patchViaSocketPath, _timeout, s_logger);
          + final Script command = new Script(_patchViaSocketPath, 5*1000, s_logger);
          String result;
          command.add("-n",vmName);
          command.add("-p", cmdLine.replaceAll(" ", "%"));
          result = command.execute();
          if (result != null) { - throw new InternalErrorException(result); + s_logger.debug("passcmd failed:" + result); }

          }

        Show
        Wei Zhou added a comment - Rayees, I notice Edison changed something in commit ba4c4400b5095e9c14e986f0cdfe6c7d1fd861a8, Could you test it again? [root@weizhou-centos master] # git show ba4c4400b5095e9c14e986f0cdfe6c7d1fd861a8 plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java commit ba4c4400b5095e9c14e986f0cdfe6c7d1fd861a8 Author: Edison Su <sudison@gmail.com> Date: Tue Jul 16 18:04:29 2013 -0700 be able to upload template into swift diff --git a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java index e51fbda..da86612 100755 — a/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java +++ b/plugins/hypervisors/kvm/src/com/cloud/hypervisor/kvm/resource/LibvirtComputingResource.java @@ -1072,13 +1072,13 @@ ServerResource { private void passCmdLine(String vmName, String cmdLine) throws InternalErrorException { final Script command = new Script(_patchViaSocketPath, _timeout, s_logger); + final Script command = new Script(_patchViaSocketPath, 5*1000, s_logger); String result; command.add("-n",vmName); command.add("-p", cmdLine.replaceAll(" ", "%")); result = command.execute(); if (result != null) { - throw new InternalErrorException(result); + s_logger.debug("passcmd failed:" + result); } }
        Rayees Namathponnan made changes -
        Attachment CLOUDSTACK-3538_Debug.rar [ 12592608 ]
        Hide
        Rayees Namathponnan added a comment -

        Please see the attached debug log (CLOUDSTACK-3538_Debug)

        In this log, i restarted VM "r-17-QA" , after that r-17-QA not coming to running state in UI, below command executing in a loop

        [root@Rack2Host12 agent]# ps aux|grep patch
        root 9677 0.0 0.0 128124 3924 ? S 13:07 0:00 /usr/bin/perl -w /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n r-17-QA -p %template=domP%name=r-17-QA%eth2ip=10.223.122.95%eth2mask=255.255.255.192%gateway=10.223.122.65%eth0ip=10.1.1.1%eth0mask=255.255.255.0%domain=cs8auto.advanced%dhcprange=10.1.1.1%eth1ip=169.254.0.186%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=8.8.8.8
        root 9927 0.0 0.0 103240 860 pts/15 R+ 13:09 0:00 grep patch

        Show
        Rayees Namathponnan added a comment - Please see the attached debug log ( CLOUDSTACK-3538 _Debug) In this log, i restarted VM "r-17-QA" , after that r-17-QA not coming to running state in UI, below command executing in a loop [root@Rack2Host12 agent] # ps aux|grep patch root 9677 0.0 0.0 128124 3924 ? S 13:07 0:00 /usr/bin/perl -w /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/patchviasocket.pl -n r-17-QA -p %template=domP%name=r-17-QA%eth2ip=10.223.122.95%eth2mask=255.255.255.192%gateway=10.223.122.65%eth0ip=10.1.1.1%eth0mask=255.255.255.0%domain=cs8auto.advanced%dhcprange=10.1.1.1%eth1ip=169.254.0.186%eth1mask=255.255.0.0%type=router%disable_rp_filter=true%dns1=8.8.8.8 root 9927 0.0 0.0 103240 860 pts/15 R+ 13:09 0:00 grep patch
        Rayees Namathponnan made changes -
        Attachment CLOUDSTACK-3538.rar [ 12592388 ]
        Hide
        Rayees Namathponnan added a comment -

        Attaching Logs

        Show
        Rayees Namathponnan added a comment - Attaching Logs
        Rayees Namathponnan made changes -
        Description Below BVT test cases failed in latest 4.2 run

        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_routers.TestRouterServices.test_08_start_router
        integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
        integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm


        All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,


        Please see the attached log after restart router "r-133-QA"
        -----------------------------------------------------------------------------------

        1) Issued stop command @ 11:42:02

        2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Execution is successful.
        2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Try to stop the vm at first
        2013-07-15 11:42:10,777 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) successfully shut down vm r-133-QA
        2013-07-15 11:42:52,454 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Ignoring VM r-133-QA in transition state stopping.
        2013-07-15 11:42:52,455 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
        2013-07-15 11:42:52,759 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Execution is successful.

        2) Started Command issue after 15 minute @ 11:58:03

        2013-07-15 11:58:03,238 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 169.254.0.187 -c
        2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 5-1532821715: { Cmd , MgmtId: 29066118877352, via: 5, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":133,"name":"r-133-QA","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-133-QA eth2ip=10.223.122.92 eth2mask=255.255.255.192 gateway=10.223.122.65 eth0ip=10.1.1.1 eth0mask=255.255.255.0 domain=cs43auto.advanced dhcprange=10.1.1.1 eth1ip=169.254.1.214 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"ee61a6681d35c765","params":{},"uuid":"240b6249-e20f-4c35-93b4-024303bef80a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c91d846-6310-4440-aa1d-67b35f2d2c36","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"fff90cb5-06dd-33b3-8815-d78c08ca01d9","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/rayees/SC_QA_AUTO4/primary","port":2049}},"name":"ROOT-133","size":147456,"path":"d4476933-b42c-45f4-a061-22c1908edca5","volumeId":138,"vmName":"r-133-QA","accountId":67,"format":"QCOW2","id":138}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"1b55ad5b-8516-4e79-90a4-33052c43b0e0","ip":"10.223.122.92","netmask":"255.255.255.192","gateway":"10.223.122.65","mac":"06:a0:90:00:00:55","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"859700c2-124b-4183-a64c-8357f57a7671","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:21:a1:00:02","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2311","isolationUri":"vlan://2311","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"4ce6c02e-6df6-4960-857c-3e5fdd2d8e1b","ip":"169.254.1.214","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:d6","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.223.50.67","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.1.214","port":3922,"interval":6,"retries":100,"name":"r-133-QA","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}},{},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":67,"publicIp":"10.223.122.92","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1221","vlanGateway":"10.223.122.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:92:84:00:00:55","networkRate":200,"trafficType":"Public"}],"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:39:b4:00:01","vmIpAddress":"10.1.1.120","vmName":"dVM","defaultRouter":"10.1.1.1","defaultDns":"10.1.1.1","duid":"00:03:00:01:02:00:39:b4:00:01","isDefault":true,"executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.1.214","router.name":"r-133-QA"},"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.120","vmName":"dVM","executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}}] }
        2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.StartCommand
        2013-07-15 11:58:03,363 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.

        3) r-133-QA still in starting state, even after 20 minute


        Also below refer below check-in this might be caused this
        https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c
        Below BVT test cases failed in latest 4.2 run

        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_routers.TestRouterServices.test_08_start_router
        integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
        integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm


        All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,


        Please see the attached log after restart router "r-133-QA"
        -----------------------------------------------------------------------------------

        1) Issued stop command @ 11:42:02

        2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Execution is successful.
        2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Try to stop the vm at first
        2013-07-15 11:42:10,777 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) successfully shut down vm r-133-QA
        2013-07-15 11:42:52,454 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Ignoring VM r-133-QA in transition state stopping.
        2013-07-15 11:42:52,455 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
        2013-07-15 11:42:52,759 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Execution is successful.

        2) Started Command issue after 15 minute @ 11:58:03

        2013-07-15 11:58:03,238 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 169.254.0.187 -c
        2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 5-1532821715: { Cmd , MgmtId: 29066118877352, via: 5, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":133,"name":"r-133-QA","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-133-QA eth2ip=10.223.122.92 eth2mask=255.255.255.192 gateway=10.223.122.65 eth0ip=10.1.1.1 eth0mask=255.255.255.0 domain=cs43auto.advanced dhcprange=10.1.1.1 eth1ip=169.254.1.214 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"ee61a6681d35c765","params":{},"uuid":"240b6249-e20f-4c35-93b4-024303bef80a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c91d846-6310-4440-aa1d-67b35f2d2c36","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"fff90cb5-06dd-33b3-8815-d78c08ca01d9","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/rayees/SC_QA_AUTO4/primary","port":2049}},"name":"ROOT-133","size":147456,"path":"d4476933-b42c-45f4-a061-22c1908edca5","volumeId":138,"vmName":"r-133-QA","accountId":67,"format":"QCOW2","id":138}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"1b55ad5b-8516-4e79-90a4-33052c43b0e0","ip":"10.223.122.92","netmask":"255.255.255.192","gateway":"10.223.122.65","mac":"06:a0:90:00:00:55","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"859700c2-124b-4183-a64c-8357f57a7671","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:21:a1:00:02","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2311","isolationUri":"vlan://2311","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"4ce6c02e-6df6-4960-857c-3e5fdd2d8e1b","ip":"169.254.1.214","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:d6","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.223.50.67","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.1.214","port":3922,"interval":6,"retries":100,"name":"r-133-QA","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}},{},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":67,"publicIp":"10.223.122.92","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1221","vlanGateway":"10.223.122.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:92:84:00:00:55","networkRate":200,"trafficType":"Public"}],"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:39:b4:00:01","vmIpAddress":"10.1.1.120","vmName":"dVM","defaultRouter":"10.1.1.1","defaultDns":"10.1.1.1","duid":"00:03:00:01:02:00:39:b4:00:01","isDefault":true,"executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.1.214","router.name":"r-133-QA"},"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.120","vmName":"dVM","executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}}] }
        2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.StartCommand
        2013-07-15 11:58:03,363 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.

        3) r-133-QA still in starting state in UI , even after 20 minute

        See the "virsh list" from the host, r-133-QA is missing

        [root@Rack2Host12 agent]# virsh list
         Id Name State
        ----------------------------------------------------
         3 r-130-QA running
         5 r-128-QA running
         6 r-126-QA running
         7 i-65-129-QA running
         10 i-71-139-QA running
         11 r-57-QA running
         13 r-144-QA running


        Also below refer below check-in this might be caused this
        https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c
        Rayees Namathponnan made changes -
        Description Below BVT test cases failed in latest 4.2 run

        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_routers.TestRouterServices.test_08_start_router
        integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
        integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm


        All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,


        Please see the attached log ()
        -----------------------------------------


        Also below refer below check-in this might be caused this
        https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c
        Below BVT test cases failed in latest 4.2 run

        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_routers.TestRouterServices.test_08_start_router
        integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
        integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm


        All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,


        Please see the attached log after restart router "r-133-QA"
        -----------------------------------------------------------------------------------

        1) Issued stop command @ 11:42:02

        2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Execution is successful.
        2013-07-15 11:42:02,765 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) Try to stop the vm at first
        2013-07-15 11:42:10,777 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-1:null) successfully shut down vm r-133-QA
        2013-07-15 11:42:52,454 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Ignoring VM r-133-QA in transition state stopping.
        2013-07-15 11:42:52,455 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Executing: /usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms
        2013-07-15 11:42:52,759 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) Execution is successful.

        2) Started Command issue after 15 minute @ 11:58:03

        2013-07-15 11:58:03,238 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /usr/share/cloudstack-common/scripts/network/domr/router_proxy.sh netusage.sh 169.254.0.187 -c
        2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 5-1532821715: { Cmd , MgmtId: 29066118877352, via: 5, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StartCommand":{"vm":{"id":133,"name":"r-133-QA","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":134217728,"maxRam":134217728,"arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP name=r-133-QA eth2ip=10.223.122.92 eth2mask=255.255.255.192 gateway=10.223.122.65 eth0ip=10.1.1.1 eth0mask=255.255.255.0 domain=cs43auto.advanced dhcprange=10.1.1.1 eth1ip=169.254.1.214 eth1mask=255.255.0.0 type=router disable_rp_filter=true dns1=8.8.8.8","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"ee61a6681d35c765","params":{},"uuid":"240b6249-e20f-4c35-93b4-024303bef80a","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"5c91d846-6310-4440-aa1d-67b35f2d2c36","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"fff90cb5-06dd-33b3-8815-d78c08ca01d9","id":1,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/rayees/SC_QA_AUTO4/primary","port":2049}},"name":"ROOT-133","size":147456,"path":"d4476933-b42c-45f4-a061-22c1908edca5","volumeId":138,"vmName":"r-133-QA","accountId":67,"format":"QCOW2","id":138}},"diskSeq":0,"type":"ROOT"}],"nics":[{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"uuid":"1b55ad5b-8516-4e79-90a4-33052c43b0e0","ip":"10.223.122.92","netmask":"255.255.255.192","gateway":"10.223.122.65","mac":"06:a0:90:00:00:55","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false},{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"uuid":"859700c2-124b-4183-a64c-8357f57a7671","ip":"10.1.1.1","netmask":"255.255.255.0","mac":"02:00:21:a1:00:02","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2311","isolationUri":"vlan://2311","isSecurityGroupEnabled":false},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"4ce6c02e-6df6-4960-857c-3e5fdd2d8e1b","ip":"169.254.1.214","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:01:d6","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false}]},"hostIp":"10.223.50.67","executeInSequence":false,"wait":0}},{"com.cloud.agent.api.check.CheckSshCommand":{"ip":"169.254.1.214","port":3922,"interval":6,"retries":100,"name":"r-133-QA","wait":0}},{"com.cloud.agent.api.GetDomRVersionCmd":{"accessDetails":{"router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}},{},{"com.cloud.agent.api.routing.IpAssocCommand":{"ipAddresses":[{"accountId":67,"publicIp":"10.223.122.92","sourceNat":true,"add":true,"oneToOneNat":false,"firstIP":true,"vlanId":"1221","vlanGateway":"10.223.122.65","vlanNetmask":"255.255.255.192","vifMacAddress":"06:92:84:00:00:55","networkRate":200,"trafficType":"Public"}],"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}},{"com.cloud.agent.api.routing.DhcpEntryCommand":{"vmMac":"02:00:39:b4:00:01","vmIpAddress":"10.1.1.120","vmName":"dVM","defaultRouter":"10.1.1.1","defaultDns":"10.1.1.1","duid":"00:03:00:01:02:00:39:b4:00:01","isDefault":true,"executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.ip":"169.254.1.214","router.name":"r-133-QA"},"wait":0}},{"com.cloud.agent.api.routing.VmDataCommand":{"vmIpAddress":"10.1.1.120","vmName":"dVM","executeInSequence":false,"accessDetails":{"router.guest.ip":"10.1.1.1","zone.network.type":"Advanced","router.name":"r-133-QA","router.ip":"169.254.1.214"},"wait":0}}] }
        2013-07-15 11:58:03,291 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.StartCommand
        2013-07-15 11:58:03,363 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.

        3) r-133-QA still in starting state, even after 20 minute


        Also below refer below check-in this might be caused this
        https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c
        Rayees Namathponnan made changes -
        Field Original Value New Value
        Description Below BVT test cases failed in latest 4.2 run

        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_routers.TestRouterServices.test_08_start_router
        integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
        integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm


        All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,


        Also below refer below check-in this might be caused this

        https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c
        Below BVT test cases failed in latest 4.2 run

        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_network.TestRebootRouter.test_reboot_router
        integration.smoke.test_routers.TestRouterServices.test_08_start_router
        integration.smoke.test_routers.TestRouterServices.test_09_reboot_router
        integration.smoke.test_ssvm.TestSSVMs.test_07_reboot_ssvm


        All test cases failed to reboot router or SSVM, also observed cloudstack taking more than 20 minute to create new router ,


        Please see the attached log ()
        -----------------------------------------


        Also below refer below check-in this might be caused this
        https://git-wip-us.apache.org/repos/asf?p=cloudstack.git;a=patch;h=22c6df0ba25f1f1fd3cf489a29ed1f4f4097c89c
        Rayees Namathponnan created issue -

          People

          • Assignee:
            edison su
            Reporter:
            Rayees Namathponnan
          • Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development