Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-9286

Delete Domain not working: Failed to clean up domain resources and sub domains, delete failed on domain

Add voteWatch issue
    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.7.0
    • None
    • Automation
    • Security Level: Public (Anyone can view this level - this is the default.)
    • None
    • Two CentOS7 MGMT Servers
      XenServer Hypervisors

    Description

      If you disable an account of a domain and then try to delete that domain it fails with following error:
      –
      2016-02-15 16:22:04,921 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635) (logid:7354b28c) Executing AsyncJobVO {id:1635, userId: 18, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd, cmdInfo: {"cleanup":"true","response":"json","ctxUserId":"18","httpmethod":"GET","ctxStartEventId":"4107","id":"dd40f585-73d6-4710-a4b7-7dd74e910675","ctxDetails":"

      {\"interface com.cloud.domain.Domain\":\"dd40f585-73d6-4710-a4b7-7dd74e910675\"}

      ","ctxAccountId":"2","uuid":"dd40f585-73d6-4710-a4b7-7dd74e910675","cmdEventType":"DOMAIN.DELETE","_":"1455549724913"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 22095312942840, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
      2016-02-15 16:22:04,921 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-11:ctx-5ea86460 ctx-69c37a64) (logid:ee5245e3) submit async job-1635, details: AsyncJobVO {id:1635, userId: 18, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd, cmdInfo: {"cleanup":"true","response":"json","ctxUserId":"18","httpmethod":"GET","ctxStartEventId":"4107","id":"dd40f585-73d6-4710-a4b7-7dd74e910675","ctxDetails":"

      {\"interface com.cloud.domain.Domain\":\"dd40f585-73d6-4710-a4b7-7dd74e910675\"}

      ","ctxAccountId":"2","uuid":"dd40f585-73d6-4710-a4b7-7dd74e910675","cmdEventType":"DOMAIN.DELETE","_":"1455549724913"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 22095312942840, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
      2016-02-15 16:22:04,922 DEBUG [c.c.a.ApiServlet] (catalina-exec-11:ctx-5ea86460 ctx-69c37a64) (logid:ee5245e3) ===END=== 212.123.98.2 – GET command=deleteDomain&id=dd40f585-73d6-4710-a4b7-7dd74e910675&cleanup=true&response=json&_=1455549724913
      2016-02-15 16:22:04,937 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c) Access granted to Acct[c1022585-b874-11e5-b4fa-1418774e8f7a-admin] to Domain:97/xxxxx/xxxxxxxxxxx/ by AffinityGroupAccessChecker
      2016-02-15 16:22:04,937 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c) Marking domain id=97 as Inactive before actually deleting it
      2016-02-15 16:22:04,940 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c) Cleaning up domain id=97
      2016-02-15 16:22:04,949 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c) Deleting networks for domain id=97
      2016-02-15 16:22:04,952 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c) Can't delete the domain yet because it has 1accounts that need a cleanup
      2016-02-15 16:22:04,952 ERROR [c.c.u.DomainManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c) Exception deleting domain with id 97
      com.cloud.utils.exception.CloudRuntimeException: Failed to clean up domain resources and sub domains, delete failed on domain 5002000011 (id: 97).
      at com.cloud.user.DomainManagerImpl.deleteDomain(DomainManagerImpl.java:268)
      at com.cloud.user.DomainManagerImpl.deleteDomain(DomainManagerImpl.java:251)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
      at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at com.sun.proxy.$Proxy128.deleteDomain(Unknown Source)
      at org.apache.cloudstack.region.RegionManagerImpl.deleteDomain(RegionManagerImpl.java:240)
      at org.apache.cloudstack.region.RegionServiceImpl.deleteDomain(RegionServiceImpl.java:167)
      at org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd.execute(DeleteDomainCmd.java:103)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
      at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      2016-02-15 16:22:04,953 DEBUG [c.c.u.DomainManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635 ctx-633df0cf) (logid:7354b28c) Changing domain id=97 state back to Active because it can't be removed due to resources referencing to it
      2016-02-15 16:22:04,962 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-25:ctx-da3cbbe6 job-1635) (logid:7354b28c) Unexpected exception while executing org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd
      com.cloud.utils.exception.CloudRuntimeException: Failed to clean up domain resources and sub domains, delete failed on domain 5002000011 (id: 97).
      at com.cloud.user.DomainManagerImpl.deleteDomain(DomainManagerImpl.java:268)
      at com.cloud.user.DomainManagerImpl.deleteDomain(DomainManagerImpl.java:251)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:497)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
      at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at com.sun.proxy.$Proxy128.deleteDomain(Unknown Source)
      at org.apache.cloudstack.region.RegionManagerImpl.deleteDomain(RegionManagerImpl.java:240)
      at org.apache.cloudstack.region.RegionServiceImpl.deleteDomain(RegionServiceImpl.java:167)
      at org.apache.cloudstack.api.command.admin.domain.DeleteDomainCmd.execute(DeleteDomainCmd.java:103)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
      at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:745)
      2016-02-15 16:22:04,962 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-25:ctx-da3cbbe6 job-1635) (logid:7354b28c) Complete async job-1635, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/

      {"uuidList":[],"errorcode":530,"errortext":"Failed to clean up domain resources and sub domains, delete failed on domain 5002000011 (id: 97)."}

      ------

      The account is not showing up in webgui anymore, so you cant enable the account via GUI.

      As workaround you can enable the account via mysql and then it will cleanup and remove successfully:
      –
      update account set state='enabled' where id="77" limit 1;
      –

      => Cleanup successfully!

      I would expect, that cloudstack automation will do this for me or it`s cleanup process will not rely on an enabled account.

      Attachments

        Activity

          People

            Unassigned Unassigned
            disappear Thomas
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:

              Slack

                Issue deployment