Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-9932

Snapshot is getting deleted while volume creation from the snapshot is in progress

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: 4.10.0.0
    • Fix Version/s: None
    • Component/s: None
    • Security Level: Public (Anyone can view this level - this is the default.)
    • Labels:
      None
    • Environment:
      ACS -4.10.
      http://10.112.3.52:8080/client/
      user : admin
      password : password

      Description

      Issue:
      -------
      Snapshot is getting deleted while the volume create request from the snapshot is in progress.
      Steps to reproduce :
      -------------------------
      1) Create root/data volume snapshot
      2) Create Volumes from the snapshot
      3) Delete the snapshot immediately while the volume creation is in progress
      Existing behavior:
      ---------------------
      Snapshot is getting deleted from the ACS and status is "Destroyed" in the snapshots table.
      Error message:
      ---------------------
      2017-05-30 04:10:17,120 DEBUG [c.c.a.t.Request] (API-Job-Executor-26:ctx-8ba0012a job-188 ctx-ffe9988e) (logid:fd7a5d83) Seq 1-1570067420092044880: Received: { Ans: , MgmtId: 196796298644167, via: 1(localhost), Ver: v1, Flags: 110,

      { CopyCmdAnswer }

      }
      2017-05-30 04:10:17,131 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188 ctx-ffe9988e) (logid:fd7a5d83) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5c07d712), no need to delete from object in store ref table
      2017-05-30 04:10:17,139 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-26:ctx-8ba0012a job-188 ctx-ffe9988e) (logid:fd7a5d83) Failed to create volume from snapshot:null due to Task failed! Task record: uuid: bf2abdea-bea3-36e2-22f0-8fa62985c614
      nameLabel: Async.VDI.copy
      nameDescription:
      allowedOperations: []
      currentOperations: {}
      created: Tue May 30 04:10:10 EDT 2017
      finished: Tue May 30 04:10:16 EDT 2017
      status: failure
      residentOn: com.xensource.xenapi.Host@7a705050
      progress: 1.0
      type: <none/>
      result:
      errorInfo: [INTERNAL_ERROR, End_of_file]
      otherConfig: {}
      subtaskOf: com.xensource.xenapi.Task@aaf13f6f
      subtasks: []

      2017-05-30 04:10:17,166 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188 ctx-ffe9988e) (logid:fd7a5d83) Access granted to Acct[def73ff3-3fbb-11e7-b95a-b2fc35794ac7-admin] to Domain:1/ by AffinityGroupAccessChecker
      2017-05-30 04:10:17,180 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-26:ctx-8ba0012a job-188) (logid:fd7a5d83) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.CreateVolumeCmdByAdmin
      com.cloud.utils.exception.CloudRuntimeException: Failed to create volume: 42
      at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:803)
      at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:175)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:498)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
      at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
      at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
      at com.sun.proxy.$Proxy207.createVolume(Unknown Source)
      at org.apache.cloudstack.api.command.admin.volume.CreateVolumeCmdByAdmin.execute(CreateVolumeCmdByAdmin.java:42)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
      at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:558)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:506)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
      at java.util.concurrent.FutureTask.run(FutureTask.java:266)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
      at java.lang.Thread.run(Thread.java:748)
      Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to create volume from snapshot:null due to Task failed! Task record: uuid: bf2abdea-bea3-36e2-22f0-8fa62985c614
      nameLabel: Async.VDI.copy
      nameDescription:
      allowedOperations: []
      currentOperations: {}
      created: Tue May 30 04:10:10 EDT 2017
      finished: Tue May 30 04:10:16 EDT 2017
      status: failure
      residentOn: com.xensource.xenapi.Host@7a705050
      progress: 1.0
      type: <none/>
      result:
      errorInfo: [INTERNAL_ERROR, End_of_file]
      otherConfig: {}
      subtaskOf: com.xensource.xenapi.Task@aaf13f6f
      subtasks: []

      at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeFromSnapshot(VolumeOrchestrator.java:439)
      at com.cloud.storage.VolumeApiServiceImpl.createVolumeFromSnapshot(VolumeApiServiceImpl.java:825)
      at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:775)
      ... 31 more
      2017-05-30 04:10:17,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188) (logid:fd7a5d83) Complete async job-188, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/

      {"uuidList":[],"errorcode":530,"errortext":"Failed to create volume: 42"}

      2017-05-30 04:10:17,194 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188) (logid:fd7a5d83) Publish async job-188 complete on message bus
      2017-05-30 04:10:17,194 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188) (logid:fd7a5d83) Wake up jobs related to job-188

      Expected Behavior:
      -------------------------
      It should not be getting deleted while the volume request is in progress. or It should give warning message or Alert message as volume create request is in progress on the snapshot.

      Below are the details from snapshot and snapshot_store_ref tables.
      ------------------------------------------------------------------------------------------
      mysql> select * from snapshots where id=9\G;

      id: 9
      data_center_id: 1
      account_id: 2
      domain_id: 1
      volume_id: 37
      disk_offering_id: 2
      status: Destroyed
      path: NULL
      name: m29snap2
      uuid: f08ff2b5-b889-41c2-bc15-ccc7b94342b8
      snapshot_type: 0
      type_description: MANUAL
      size: 21474836480
      created: 2017-05-29 10:33:13
      removed: NULL
      backup_snap_id: NULL
      swift_id: NULL
      sechost_id: NULL
      prev_snap_id: NULL
      hypervisor_type: XenServer
      version: 2.2
      s3_id: NULL
      min_iops: NULL
      max_iops: NULL
      location_type: NULL
      1 row in set (0.00 sec)

      mysql> select * from snapshot_store_ref where snapshot_id=9\G;

      id: 17
      store_id: 1
      snapshot_id: 9
      created: 2017-05-29 10:33:13
      last_updated: NULL
      job_id: NULL
      store_role: Primary
      size: 21474836480
      physical_size: 21474836480
      parent_snapshot_id: 0
      install_path: bf345178-07f7-46c8-ab50-e5ecce67c1ba
      state: Destroyed
      update_count: 2
      ref_cnt: 0
      updated: 2017-05-29 10:33:17
      volume_id: 37
      ==============================================

      id: 18
      store_id: 1
      snapshot_id: 9
      created: 2017-05-29 10:33:17
      last_updated: NULL
      job_id: NULL
      store_role: Image
      size: 21474836480
      physical_size: 45056
      parent_snapshot_id: 0
      install_path: snapshots/2/37/8e0f004e-f916-4f4f-895d-8d95eab00463.vhd
      state: Destroyed
      update_count: 4
      ref_cnt: 0
      updated: 2017-05-30 08:10:15
      volume_id: 37
      2 rows in set (0.00 sec)

      [root@localhost primary]# ls bf345178-07f7-46c8-ab50-e5ecce67c1ba.vhd
      bf345178-07f7-46c8-ab50-e5ecce67c1ba.vhd

      [root@localhost secondary]# cd snapshots/
      [root@localhost snapshots]# cd 2/37/
      [root@localhost 37]# ls

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                pavan4smile Pavan Kumar Aravapalli
                Reporter:
                veerapuppala veerapuppala
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: