Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-1007

Not able to delete Shared network because of not being able to stop the router.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Major
    • Resolution: Not A Problem
    • 4.1.0, 4.2.0
    • 4.4.3
    • Management Server
    • Security Level: Public (Anyone can view this level - this is the default.)
    • None
    • Build from "network-refactor"

    Description

      Steps to reproduce the problem:

      Set up - Advanced zone with KVM host.

      Create a Zone wide Shared network.

      Deploy Vm using this network.

      Destroy this Vm.

      Wait for the Vm to get Expunged.

      Delete the network.

      Network deletion fails because we are not able to destroy the router.

      Management server logs:

      2013-01-17 18:05:31,818 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-6:job-34) Sending destroy to com.cloud.network.element.VirtualRouterElement$$EnhancerByCGLIB$$d659f5e7@447d4275
      2013-01-17 18:05:31,820 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (Job-Executor-6:job-34) Attempting to destroy router 10
      2013-01-17 18:05:31,840 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-6:job-34) VM state transitted from :Running to Stopping with event: StopRequestedvm's original host id: 1 new host id: 1 host id before state transition: 1
      2013-01-17 18:05:31,850 DEBUG [agent.transport.Request] (Job-Executor-6:job-34) Seq 1-772669469: Sending { Cmd , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-10-VM","wait":0}}] }
      2013-01-17 18:05:33,284 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) HostStatsCollector is running...
      2013-01-17 18:05:33,883 DEBUG [agent.transport.Request] (StatsCollector-2:null) Seq 1-772669470: Received: { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 10,

      { GetHostStatsAnswer }

      }
      2013-01-17 18:05:36,715 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===START=== 10.217.252.59 – GET command=queryAsyncJobResult&jobId=82e583fa-7829-4396-a2f7-4358e2abba80&response=json&sessionkey=hd3IkmVY5KODN5MiyOXg%2FapVTb4%3D&_=1358464170203
      2013-01-17 18:05:36,746 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) ===END=== 10.217.252.59 – GET command=queryAsyncJobResult&jobId=82e583fa-7829-4396-a2f7-4358e2abba80&response=json&sessionkey=hd3IkmVY5KODN5MiyOXg%2FapVTb4%3D&_=1358464170203
      2013-01-17 18:05:38,232 DEBUG [agent.transport.Request] (AgentManager-Handler-15:null) Seq 1-772669469: Processing: { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110, [{"Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStorageAdaptor(KVMStoragePoolManager.java:41)\n\tat com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:66)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3153)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2687)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:968)\n\tat com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat com.cloud.utils.nio.Task.run(Task.java:83)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
      2013-01-17 18:05:38,232 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-15:null) Seq 1-772669469: No more commands found
      2013-01-17 18:05:38,232 DEBUG [agent.transport.Request] (Job-Executor-6:job-34) Seq 1-772669469: Received: { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110,

      { Answer }

      }
      2013-01-17 18:05:38,232 WARN [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-34) Unable to stop vm VM[DomainRouter|r-10-VM]
      2013-01-17 18:05:38,299 DEBUG [cloud.capacity.CapacityManagerImpl] (Job-Executor-6:job-34) VM state transitted from :Stopping to Running with event: OperationFailedvm's original host id: 1 new host id: 1 host id before state transition: 1
      2013-01-17 18:05:38,300 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-34) Unable to stop the VM so we can't expunge it.
      2013-01-17 18:05:38,300 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-34) Unable to destroy the vm because it is not in the correct state: VM[DomainRouter|r-10-VM]
      2013-01-17 18:05:38,300 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-6:job-34) Did not expunge VM[DomainRouter|r-10-VM]
      2013-01-17 18:05:38,300 WARN [cloud.network.NetworkManagerImpl] (Job-Executor-6:job-34) Unable to complete destroy of the network: failed to destroy network element VirtualRouter
      2013-01-17 18:05:38,317 WARN [cloud.api.ApiDispatcher] (Job-Executor-6:job-34) class com.cloud.api.ServerApiException : Failed to delete network
      2013-01-17 18:05:38,318 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-6:job-34) Complete async job-34, jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to delete network
      2013-01-17 18:05:38,352 DEBUG [cloud.async.SyncQueueManagerImpl] (Job-Executor-6:job-34) Sync queue (10) is currently empty
      2013-01-17 18:05:38,352 WARN [cloud.async.AsyncJobManagerImpl] (Job-Executor-6:job-34) Unable to unregister active job 34 from JMX monitoring
      2013-01-17 18:05:41,714 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null) ===START=== 10.217.252.59 – GET command=queryAsyncJobResult&jobId=82e583fa-7829-4396-a2f7-4358e2abba80&response=json&sessionkey=hd3IkmVY5KODN5MiyOXg%2FapVTb4%3D&_=1358464175201
      2013-01-17 18:05:41,728 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-8:null) Async job-34 completed

      in agent.log:

      013-01-17 23:09:38,641 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Request:Seq 1-772669469: { Cmd , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 100111, [{"StopCommand":{"isProxy":false,"vmName":"r-10-VM","wait":0}}] }
      2013-01-17 23:09:38,641 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Processing command: com.cloud.agent.api.StopCommand
      2013-01-17 23:09:38,703 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) Executing: /usr/lib64/cloud/common/scripts/vm/network/security_group.py destroy_network_rules_for_vm --vmname r-10-VM --vif vnet12
      2013-01-17 23:09:38,857 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) Execution is successful.
      2013-01-17 23:09:38,858 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-2:null) Try to stop the vm at first
      2013-01-17 23:09:40,083 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing command: com.cloud.agent.api.GetHostStatsCommand
      2013-01-17 23:09:40,083 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /bin/bash -c idle=$(top -b -n 1|grep Cpu(s):|cut -d% -f4|cut -d, -f2);echo $idle
      2013-01-17 23:09:40,612 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.
      2013-01-17 23:09:40,613 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /bin/bash -c freeMem=$(free|grep cache:|awk '

      {print $4}

      ');echo $freeMem
      2013-01-17 23:09:40,627 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.
      2013-01-17 23:09:40,627 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /bin/bash -c free|grep Mem:|awk '

      {print $2}

      '
      2013-01-17 23:09:40,644 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.
      2013-01-17 23:09:40,645 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /bin/bash -c cat /sys/class/net/cloudbr0/statistics/rx_bytes
      2013-01-17 23:09:40,656 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.
      2013-01-17 23:09:40,657 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Executing: /bin/bash -c cat /sys/class/net/cloudbr0/statistics/tx_bytes
      2013-01-17 23:09:40,668 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Execution is successful.
      2013-01-17 23:09:45,014 WARN [cloud.agent.Agent] (agentRequest-Handler-2:null) Caught:
      java.lang.NullPointerException
      at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStorageAdaptor(KVMStoragePoolManager.java:41)
      at com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:66)
      at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3153)
      at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2687)
      at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:968)
      at com.cloud.agent.Agent.processRequest(Agent.java:525)
      at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)
      at com.cloud.utils.nio.Task.run(Task.java:83)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)
      2013-01-17 23:09:45,017 DEBUG [cloud.agent.Agent] (agentRequest-Handler-2:null) Seq 1-772669469: { Ans: , MgmtId: 206915885081428, via: 1, Ver: v1, Flags: 110, [{"Answer":{"result":false,"details":"java.lang.NullPointerException\n\tat com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStorageAdaptor(KVMStoragePoolManager.java:41)\n\tat com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:66)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.cleanupDisk(LibvirtComputingResource.java:3153)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:2687)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:968)\n\tat com.cloud.agent.Agent.processRequest(Agent.java:525)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:852)\n\tat com.cloud.utils.nio.Task.run(Task.java:83)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }

      Attachments

        1. agent.log
          876 kB
          Sangeetha Hariharan
        2. agent.log.2013-01-17.gz
          123 kB
          Sangeetha Hariharan
        3. management-server.zip
          496 kB
          Chandan Purushothama

        Activity

          People

            sangeethah Sangeetha Hariharan
            sangeethah Sangeetha Hariharan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: