[CLOUDSTACK-5002] unable to destroy vm ;VM destroy failed in Stop i-2-59-VM Command due to You gave an invalid object reference. The object may have recently been deleted.

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.2.1
• Fix Version/s: 4.3.0
• Component/s: Management Server, XenServer
• Security Level: Public (Anyone can view this level - this is the default.)
• Assignee: Koushik Das
• Reporter: prashant kumar mishra
• Created: 30/Oct/13 10:05
• Updated: 10/Feb/14 11:24
• Resolved: 06/Jan/14 13:07

Description

steps to reproduce
---------------------------
1-prepare CS setup with xen 6.1
2-set execute.in.sequence.hypervisor.commands and execute.in.sequence.network.element.commands to false
3-Restart MS
4-deploy 35 vms parallelly with default centOS template
5-try to destroy all uservms using script

Expected
-------------
All vms should get destroyed

Actual
------------
7 vms went in stopped state

Logs
-------
2013-10-30 10:22:35,406 DEBUG [agent.transport.Request] (Job-Executor-93:job-244 = [ 539bf83c-67bb-4d4a-a8e2-837a8d85cb4d ]) Seq 3-125567535: Sending { Cmd , MgmtId: 7484181839895, via: 3, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-59-VM","wait":0}}] }
2013-10-30 10:22:35,406 DEBUG [agent.transport.Request] (Job-Executor-93:job-244 = [ 539bf83c-67bb-4d4a-a8e2-837a8d85cb4d ]) Seq 3-125567535: Executing: { Cmd , MgmtId: 7484181839895, via: 3, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-59-VM","wait":0}}] }

2013-10-30 10:22:36,050 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-68:null) 9. The VM i-2-59-VM is in Stopping state

013-10-30 10:22:39,747 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-490:null) VM i-2-59-VM: cs state = Stopping and realState = Running
2013-10-30 10:22:39,747 DEBUG [cloud.vm.VirtualMachineManagerImpl] (DirectAgent-490:null) VM i-2-59-VM: cs state = Stopping and realState = Running

2013-10-30 10:32:36,831 WARN [xen.resource.CitrixResourceBase] (DirectAgent-68:null) Async 600 seconds timeout for task com.xensource.xenapi.Task@53bab0c6
2013-10-30 10:32:36,843 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-68:null) Unable to cleanShutdown VM(i-2-59-VM) on host(7f8a4f8f-1c15-4b7a-95cf-1f64b09ce674) due to Async 600 seconds timeout for task com.xensource.xenapi.Task@53bab0c6

2013-10-30 10:42:05,852 DEBUG [agent.transport.Request] (HA-Worker-4:work-40) Seq 3-125567622: Sending { Cmd , MgmtId: 7484181839895, via: 3, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-59-VM","wait":0}}] }
2013-10-30 10:42:05,853 DEBUG [agent.transport.Request] (HA-Worker-4:work-40) Seq 3-125567622: Executing: { Cmd , MgmtId: 7484181839895, via: 3, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":false,"vmName":"i-2-59-VM","wait":0}}] }

013-10-30 10:42:05,991 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-218:null) 9. The VM i-2-59-VM is in Stopping state
2013-10-30 10:42:05,997 WARN [xen.resource.CitrixResourceBase] (DirectAgent-152:null) VM destroy failed in Stop i-2-67-VM Command due to You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:209)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool.java:909)
at com.xensource.xenapi.VM.getPowerState(VM.java:746)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:4011)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:500)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)

2013-10-30 10:42:06,122 WARN [xen.resource.CitrixResourceBase] (DirectAgent-68:null) VM destroy failed in Stop i-2-59-VM Command due to You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of reference given, and the handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:209)

2013-10-30 10:42:06,127 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-68:null) 10. The VM i-2-59-VM is in Running state
2013-10-30 10:42:06,127 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-68:null) Seq 3-125567535: Response Received: