Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
4.10.0.0
-
None
-
None
-
Security Level: Public (Anyone can view this level - this is the default.)
-
None
-
Description
Issue:
-------
Snapshot is getting deleted while the volume create request from the snapshot is in progress.
Steps to reproduce :
-------------------------
1) Create root/data volume snapshot
2) Create Volumes from the snapshot
3) Delete the snapshot immediately while the volume creation is in progress
Existing behavior:
---------------------
Snapshot is getting deleted from the ACS and status is "Destroyed" in the snapshots table.
Error message:
---------------------
2017-05-30 04:10:17,120 DEBUG [c.c.a.t.Request] (API-Job-Executor-26:ctx-8ba0012a job-188 ctx-ffe9988e) (logid:fd7a5d83) Seq 1-1570067420092044880: Received: { Ans: , MgmtId: 196796298644167, via: 1(localhost), Ver: v1, Flags: 110,
}
2017-05-30 04:10:17,131 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188 ctx-ffe9988e) (logid:fd7a5d83) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5c07d712), no need to delete from object in store ref table
2017-05-30 04:10:17,139 DEBUG [o.a.c.e.o.VolumeOrchestrator] (API-Job-Executor-26:ctx-8ba0012a job-188 ctx-ffe9988e) (logid:fd7a5d83) Failed to create volume from snapshot:null due to Task failed! Task record: uuid: bf2abdea-bea3-36e2-22f0-8fa62985c614
nameLabel: Async.VDI.copy
nameDescription:
allowedOperations: []
currentOperations: {}
created: Tue May 30 04:10:10 EDT 2017
finished: Tue May 30 04:10:16 EDT 2017
status: failure
residentOn: com.xensource.xenapi.Host@7a705050
progress: 1.0
type: <none/>
result:
errorInfo: [INTERNAL_ERROR, End_of_file]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
2017-05-30 04:10:17,166 DEBUG [c.c.u.AccountManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188 ctx-ffe9988e) (logid:fd7a5d83) Access granted to Acct[def73ff3-3fbb-11e7-b95a-b2fc35794ac7-admin] to Domain:1/ by AffinityGroupAccessChecker
2017-05-30 04:10:17,180 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-26:ctx-8ba0012a job-188) (logid:fd7a5d83) Unexpected exception while executing org.apache.cloudstack.api.command.admin.volume.CreateVolumeCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: Failed to create volume: 42
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:803)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:175)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:333)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:190)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:157)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:107)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:168)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:213)
at com.sun.proxy.$Proxy207.createVolume(Unknown Source)
at org.apache.cloudstack.api.command.admin.volume.CreateVolumeCmdByAdmin.execute(CreateVolumeCmdByAdmin.java:42)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:558)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:506)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to create volume from snapshot:null due to Task failed! Task record: uuid: bf2abdea-bea3-36e2-22f0-8fa62985c614
nameLabel: Async.VDI.copy
nameDescription:
allowedOperations: []
currentOperations: {}
created: Tue May 30 04:10:10 EDT 2017
finished: Tue May 30 04:10:16 EDT 2017
status: failure
residentOn: com.xensource.xenapi.Host@7a705050
progress: 1.0
type: <none/>
result:
errorInfo: [INTERNAL_ERROR, End_of_file]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeFromSnapshot(VolumeOrchestrator.java:439)
at com.cloud.storage.VolumeApiServiceImpl.createVolumeFromSnapshot(VolumeApiServiceImpl.java:825)
at com.cloud.storage.VolumeApiServiceImpl.createVolume(VolumeApiServiceImpl.java:775)
... 31 more
2017-05-30 04:10:17,181 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188) (logid:fd7a5d83) Complete async job-188, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/
2017-05-30 04:10:17,194 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188) (logid:fd7a5d83) Publish async job-188 complete on message bus
2017-05-30 04:10:17,194 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-26:ctx-8ba0012a job-188) (logid:fd7a5d83) Wake up jobs related to job-188
Expected Behavior:
-------------------------
It should not be getting deleted while the volume request is in progress. or It should give warning message or Alert message as volume create request is in progress on the snapshot.
Below are the details from snapshot and snapshot_store_ref tables.
------------------------------------------------------------------------------------------
mysql> select * from snapshots where id=9\G;
id: 9
data_center_id: 1
account_id: 2
domain_id: 1
volume_id: 37
disk_offering_id: 2
status: Destroyed
path: NULL
name: m29snap2
uuid: f08ff2b5-b889-41c2-bc15-ccc7b94342b8
snapshot_type: 0
type_description: MANUAL
size: 21474836480
created: 2017-05-29 10:33:13
removed: NULL
backup_snap_id: NULL
swift_id: NULL
sechost_id: NULL
prev_snap_id: NULL
hypervisor_type: XenServer
version: 2.2
s3_id: NULL
min_iops: NULL
max_iops: NULL
location_type: NULL
1 row in set (0.00 sec)
mysql> select * from snapshot_store_ref where snapshot_id=9\G;
id: 17
store_id: 1
snapshot_id: 9
created: 2017-05-29 10:33:13
last_updated: NULL
job_id: NULL
store_role: Primary
size: 21474836480
physical_size: 21474836480
parent_snapshot_id: 0
install_path: bf345178-07f7-46c8-ab50-e5ecce67c1ba
state: Destroyed
update_count: 2
ref_cnt: 0
updated: 2017-05-29 10:33:17
volume_id: 37
==============================================
id: 18
store_id: 1
snapshot_id: 9
created: 2017-05-29 10:33:17
last_updated: NULL
job_id: NULL
store_role: Image
size: 21474836480
physical_size: 45056
parent_snapshot_id: 0
install_path: snapshots/2/37/8e0f004e-f916-4f4f-895d-8d95eab00463.vhd
state: Destroyed
update_count: 4
ref_cnt: 0
updated: 2017-05-30 08:10:15
volume_id: 37
2 rows in set (0.00 sec)
[root@localhost primary]# ls bf345178-07f7-46c8-ab50-e5ecce67c1ba.vhd
bf345178-07f7-46c8-ab50-e5ecce67c1ba.vhd
[root@localhost secondary]# cd snapshots/
[root@localhost snapshots]# cd 2/37/
[root@localhost 37]# ls
Attachments
Issue Links
- links to