Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-5395

When backup snapshot fails becasue of backup.snapshot.wait time exceeding , the vhd entries form the primary store is not getting cleared.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Won't Fix
    • 4.3.0
    • 4.4.0, 4.5.0
    • Management Server
    • Security Level: Public (Anyone can view this level - this is the default.)
    • None
    • Build from 4.3

    Description

      Steps to reproduce the problem:

      Deploy 5 Vms in each of the hosts with 10 GB , so we start with 10 Vms.
      We will be constantly writing to the ROOT volume.

      Change the backup.snapshot.wait to 10 mts and restart management server.

      Start concurrent snapshots for ROOT volumes of all the Vms.

      After 10 mts , the snapshots fail. They are present in the database in "CreatedOnPrimary" state.

      Vhd entries from the primary store fails to be cleaned up once the backsnapshpot job fails.

      In this case ,API fails with - "Failed to create snapshot due to an internal error creating snapshot for volume 81" . We should be able to present a more meaningful error message in this case.

      Expected Behavior:
      We should be able to clean up the Vhd entries from the primary store when the backsnapshpot job fails.

      In such cases , instead of
      select * from snapshot_store_ref;

      702 | 1 | 355 | 2013-12-06 01:25:43 | NULL | NULL | Primary | 0 | 0 | 0 | 2eedb23e-6c3f-4cae-832b-8ddb67c1fc60 | Ready | 2 | 0 | 2013-12-06 01:25:44 | 81 |

      703 1 356 2013-12-06 01:25:43 NULL NULL Primary 0 0 0 9d88bc01-9406-41ad-a134-e74dc1457954 Ready 2 0 2013-12-06 01:26:12 80
      704 1 357 2013-12-06 01:25:43 NULL NULL Primary 0 0 0 2667f2bc-6086-4ec3-a88d-20811eabde91 Ready 2 0 2013-12-06 01:26:08 79
      705 1 358 2013-12-06 01:25:44 NULL NULL Primary 0 0 0 522b2296-6960-46f2-af7d-10ddfbede1da Ready 2 0 2013-12-06 01:26:45 78
      706 1 359 2013-12-06 01:25:44 NULL NULL Primary 0 0 0 3b94fa9d-a5a5-4441-8f9f-275dcef90368 Ready 2 0 2013-12-06 01:26:04 77
      707 1 360 2013-12-06 01:25:44 NULL NULL Primary 0 0 0 1ec1d5ef-177f-4da4-8464-f0c6d71a4e84 Ready 2 0 2013-12-06 01:25:59 76
      708 1 361 2013-12-06 01:25:44 NULL NULL Primary 0 0 0 324e7552-b42a-4660-90d6-62015a7a478e Ready 2 0 2013-12-06 01:26:21 75
      709 1 362 2013-12-06 01:25:44 NULL NULL Primary 0 0 0 65bd522c-c2c8-471a-be37-095558d058f2 Ready 2 0 2013-12-06 01:26:16 74
      710 1 363 2013-12-06 01:25:44 NULL NULL Primary 0 0 0 d45ca6c7-7284-4150-907c-9499e9737c47 Ready 2 0 2013-12-06 01:25:46 73
      711 1 364 2013-12-06 01:25:44 NULL NULL Primary 0 0 0 4422f362-0be5-4a10-b172-45678d56f807 Ready 2 0 2013-12-06 01:25:55 72
      712 1 365 2013-12-06 01:25:44 NULL NULL Primary 0 0 0 89ffd430-3c03-45d2-9c48-9384636b9cd8 Ready 2 0 2013-12-06 01:26:01 71
      714 1 366 2013-12-06 01:25:45 NULL NULL Primary 0 0 0 fca5545c-9b83-4bc1-abd2-dd1bc82b23bd Ready 2 0 2013-12-06 01:25:53 70
      715 1 367 2013-12-06 01:25:45 NULL NULL Primary 0 0 0 033d8f55-8895-40b8-a120-11b28fa1f96e Ready 2 0 2013-12-06 01:25:50 69
      716 1 368 2013-12-06 01:25:45 NULL NULL Primary 0 0 0 e4d02558-28c2-474e-a379-970b22f33f55 Ready 2 0 2013-12-06 01:26:23 68
      717 1 369 2013-12-06 01:25:45 NULL NULL Primary 0 0 0 6f7c1ca0-9877-44af-9f77-4db7b8efc934 Ready 2 0 2013-12-06 01:26:37 67
      718 1 370 2013-12-06 01:25:45 NULL NULL Primary 0 0 0 e3860504-7e37-416a-8216-c4a021f5e8bf Ready 2 0 2013-12-06 01:25:47 66
      719 1 371 2013-12-06 01:25:45 NULL NULL Primary 0 0 0 48c2a9bd-e7ce-429f-be73-ef9c19964ad3 Ready 2 0 2013-12-06 01:26:17 65
      720 1 372 2013-12-06 01:25:45 NULL NULL Primary 0 0 0 3eabb081-d0d9-41d6-9c11-00e002031461 Ready 2 0 2013-12-06 01:25:51 64
      721 1 373 2013-12-06 01:25:46 NULL NULL Primary 0 0 0 c9968fbb-0876-4846-94aa-e44ea7e3143d Ready 2 0 2013-12-06 01:26:06 63
      723 1 374 2013-12-06 01:25:46 NULL NULL Primary 0 0 0 72fdec4a-c2f1-48e1-80bf-806b6db46820 Ready 2 0 2013-12-06 01:26:02 62
      742 1 375 2013-12-06 01:40:43 NULL NULL Primary 0 0 0 8a692b5a-569e-473c-83c4-dc32c09c8491 Ready 2 0 2013-12-06 01:40:47 60

      ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+

      2013-12-05 20:51:41,212 DEBUG [c.c.s.s.SnapshotManagerImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Failed to create snapshot
      com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
      at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
      at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
      at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy161.takeSnapshot(Unknown Source)
      at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
      at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
      at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy233.takeSnapshot(Unknown Source)
      at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
      at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
      at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)
      2013-12-05 20:51:41,217 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-42:ctx-c20be5fc ctx-5f8cfbf3) Take snapshot: 60 failed
      com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
      at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
      at sun.reflect.GeneratedMethodAccessor405.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy161.takeSnapshot(Unknown Source)
      at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
      at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
      at sun.reflect.GeneratedMethodAccessor404.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy233.takeSnapshot(Unknown Source)
      at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
      at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
      at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@8b2ca24f
      at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
      at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy161.takeSnapshot(Unknown Source)
      at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
      at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy233.takeSnapshot(Unknown Source)
      at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
      at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
      at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)
      2013-12-05 20:35:50,550 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-22:ctx-ae0b13d4 ctx-1bc38e81) Take snapshot: 81 failed
      com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
      at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:972)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy161.takeSnapshot(Unknown Source)
      at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1341)
      at com.cloud.storage.VolumeApiServiceImpl.takeSnapshot(VolumeApiServiceImpl.java:1408)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:601)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy233.takeSnapshot(Unknown Source)
      at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:181)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
      at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
      at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:520)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:722)
      Caused by: com.cloud.utils.exception.CloudRuntimeException: BackupSnapshot Failed due to Async 600 seconds timeout for task com.xensource.xenapi.Task@db9844a9
      at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:275)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:135)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:294)
      at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:951)
      ... 44 more
      2013-12-05 20:35:50,552 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-22:ctx-ae0b13d4) Complete async job-697, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/

      {"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 81"}

      2013-12-05 20:35:50,557 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-22:ctx-ae0b13d4) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-697

      Attachments

        1. 1205logs.rar
          2.32 MB
          Sangeetha Hariharan

        Issue Links

          duplicates

          Bug - A problem which impairs or prevents the functions of the product. CLOUDSTACK-5446 KVM-Secondary Store down-Even after secondary store is brought back up after being down for few hours,snapshot jobs do not get triggered with reason "there is other active snapshot tasks on the instance to which the volume is attached"

          • Major - Major loss of function.
          • Resolved

          Activity

            People

              anthonyxu Anthony Xu
              sangeethah Sangeetha Hariharan
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: