[CLOUDSTACK-9580] Unexpected exception while deleting vms concurrently

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.6.0
• Security Level: Public (Anyone can view this level - this is the default.)
• Labels: controller management network server
• Environment: esx-5.5
• Assignee: Unassigned
• Reporter: Bhavin
• Created: 08/Nov/16 02:07
• Updated: 08/Nov/16 21:45

Description

Unexpected exception while deleting 100 vms concurrently .This is a generic issue where we used Nuage as a Service provider
steps to reproduce
1. create 1 isolated networks, create 100 vms in the network
2. delete 100 vms in that network with expunge=true

expected results:
it should not throw exception and all vms should be deleted
actual results:
it throws unexpected exception and some vms dont start are stuck in expunging state

2016-10-19 15:39:08,277 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-14:ctx-86f2f5c3 job-4153) (logid:fbe038ed) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.DestroyVMCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Failed to expunge vm VM[User|i-2-405-VM]
        at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.java:2532)
        at sun.reflect.GeneratedMethodAccessor166.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
        at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
        at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
        at com.sun.proxy.$Proxy199.destroyVm(Unknown Source)