Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
4.7.0
-
None
-
Security Level: Public (Anyone can view this level - this is the default.)
-
None
-
CentOS 7, Cloudstack 4.7, Ceph V9.2.0 INFERNALIS.
Description
Failed to create snapshot with Swift on KVM
Steps to Reproduce:
=================
1.Bring up CS with KVM cluster using Swift(radosgw) for secondary, NFS for secondary staging, and rbd for primary
2.Deploy guest vm using default cent os template with both root and data disk
3.Create snapshot on root/data disk
Result:
=====
Snapshot creation failed with CloudRuntimeException: unsupported protocol
============
Snapshot is created on secondary staging storage but failed to copy to secondary storage.
Following is the log snippet:
2016-02-20 10:25:46,977 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-13:ctx-4728df07 ctx-3a589bea) (logid:e2d53208) submit async job-2728, details: AsyncJobVO {id:2728, userId: 10, accountId: 7, instanceType: Snapshot, instanceId: 19, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"7","uuid":"09a17747-dc55-438f-83b0-49538c1eef78","cmdEventType":"SNAPSHOT.CREATE","response":"json","ctxUserId":"10","volumeId":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","name":"test","ctxStartEventId":"7582","id":"19","ctxDetails":"{\"interface com.cloud.storage.Snapshot\":\"09a17747-dc55-438f-83b0-49538c1eef78\",\"interface com.cloud.storage.Volume\":\"9ddc9e9a-e98a-4049-b5e8-59f670e3f077\"}","_":"1455953146647"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90212275930085, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2016-02-20 10:25:46,979 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-4728df07 ctx-3a589bea) (logid:e2d53208) ===END=== 10.30.10.41 -- GET command=createSnapshot&response=json&volumeId=9ddc9e9a-e98a-4049-b5e8-59f670e3f077&quiescevm=false&name=test&_=1455953146647 2016-02-20 10:25:46,982 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-22ae7862 job-2728) (logid:a5032eea) Executing AsyncJobVO {id:2728, userId: 10, accountId: 7, instanceType: Snapshot, instanceId: 19, cmd: org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd, cmdInfo: {"quiescevm":"false","httpmethod":"GET","ctxAccountId":"7","uuid":"09a17747-dc55-438f-83b0-49538c1eef78","cmdEventType":"SNAPSHOT.CREATE","response":"json","ctxUserId":"10","volumeId":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","name":"test","ctxStartEventId":"7582","id":"19","ctxDetails":"{\"interface com.cloud.storage.Snapshot\":\"09a17747-dc55-438f-83b0-49538c1eef78\",\"interface com.cloud.storage.Volume\":\"9ddc9e9a-e98a-4049-b5e8-59f670e3f077\"}","_":"1455953146647"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90212275930085, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2016-02-20 10:25:48,127 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Executing AsyncJobVO {id:2731, userId: 10, accountId: 7, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAARaAAlxdWllc2NlVm1MAAhwb2xpY3lJZHQAEExqYXZhL2xhbmcvTG9uZztMAApzbmFwc2hvdElkcQB-AAFMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAcAAAAAAAAACgAAAAAAAABpdAAUVm9sdW1lQXBpU2VydmljZUltcGwAc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAAAc3EAfgAGAAAAAAAAABNzcQB-AAYAAAAAAAABfg, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90212275930085, completeMsid: null, lastUpdated: null, lastPolled: null, created: Sat Feb 20 10:25:47 MSK 2016} 2016-02-20 10:25:48,128 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Run VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 105, job origin: 2728 2016-02-20 10:25:48,137 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Execute VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot{"volumeId":382,"policyId":0,"snapshotId":19,"quiesceVm":false,"userId":10,"accountId":7,"vmId":105,"handlerName":"VolumeApiServiceImpl"} 2016-02-20 10:25:48,407 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CreateObjectCommand 2016-02-20 10:25:48,407 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CreateObjectCommand 2016-02-20 10:25:48,423 DEBUG [c.c.a.t.Request] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Seq 121-7199004004351743164: Sending { Cmd , MgmtId: 90212275930085, via: 121(srv-kvm01.cloud.bstelecom.ru), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CreateObjectCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"volume":{"uuid":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f5efdbfb-8fe7-3dfd-bc07-5088222d8b84","id":52,"poolType":"RBD","host":"ceph-mon.cloud.bstelecom.ru","path":"rbd-cl","port":6789,"url":"RBD://ceph-mon.cloud.bstelecom.ru/rbd-cl/?ROLE=Primary&STOREUUID=f5efdbfb-8fe7-3dfd-bc07-5088222d8b84"}},"name":"ROOT-105","size":472312242,"path":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeId":382,"vmName":"i-7-105-VM","accountId":7,"format":"RAW","provisioningType":"THIN","id":382,"deviceId":0,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f5efdbfb-8fe7-3dfd-bc07-5088222d8b84","id":52,"poolType":"RBD","host":"ceph-mon.cloud.bstelecom.ru","path":"rbd-cl","port":6789,"url":"RBD://ceph-mon.cloud.bstelecom.ru/rbd-cl/?ROLE=Primary&STOREUUID=f5efdbfb-8fe7-3dfd-bc07-5088222d8b84"}},"vmName":"i-7-105-VM","name":"test","hypervisorType":"KVM","id":19,"quiescevm":false,"physicalSize":0}},"wait":0}}] } 2016-02-20 10:25:49,601 DEBUG [c.c.a.t.Request] (AgentManager-Handler-5:null) (logid:) Seq 121-7199004004351743164: Processing: { Ans: , MgmtId: 90212275930085, via: 121, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CreateObjectAnswer":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"rbd-cl/9ddc9e9a-e98a-4049-b5e8-59f670e3f077/779fb0f9-dcb6-4451-b270-81f01bb7d326","id":0,"quiescevm":false,"physicalSize":0}},"result":true,"wait":0}}] } 2016-02-20 10:25:49,601 DEBUG [c.c.a.t.Request] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Seq 121-7199004004351743164: Received: { Ans: , MgmtId: 90212275930085, via: 121(srv-kvm01.cloud.bstelecom.ru), Ver: v1, Flags: 10, { CreateObjectAnswer } } 2016-02-20 10:25:49,621 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) (logid:) SeqA 142-21971: Processing Seq 142-21971: { Cmd , MgmtId: -1, via: 142, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":104,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] } 2016-02-20 10:25:49,630 DEBUG [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-3:null) (logid:) SeqA 142-21971: Sending Seq 142-21971: { Ans: , MgmtId: 90212275930085, via: 142, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] } 2016-02-20 10:25:49,794 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type SNAPSHOT 2016-02-20 10:25:49,886 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) needCacheStorage true, dest at snapshots/7/382 dest role Imagerbd-cl/9ddc9e9a-e98a-4049-b5e8-59f670e3f077/779fb0f9-dcb6-4451-b270-81f01bb7d326 src role Primary 2016-02-20 10:25:50,000 DEBUG [c.c.a.ApiServlet] (catalina-exec-9:ctx-a8490da0) (logid:6886e205) ===START=== 10.30.10.41 -- GET command=queryAsyncJobResult&jobId=a5032eea-7fdc-4f79-a613-1a7822dfabaa&response=json&_=1455953150153 2016-02-20 10:25:50,060 DEBUG [c.c.u.AccountManagerImpl] (catalina-exec-9:ctx-a8490da0 ctx-6373cd53) (logid:6886e205) Access to Acct[69296515-f19c-407c-859e-42f4599e4653-ykarpel] granted to Acct[69296515-f19c-407c-859e-42f4599e4653-ykarpel] by DomainChecker 2016-02-20 10:25:50,086 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand 2016-02-20 10:25:50,086 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) getCommandHostDelegation: class org.apache.cloudstack.storage.command.CopyCommand 2016-02-20 10:25:50,091 DEBUG [c.c.a.t.Request] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Seq 121-7199004004351743165: Sending { Cmd , MgmtId: 90212275930085, via: 121(srv-kvm01.cloud.bstelecom.ru), Ver: v1, Flags: 100111, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"rbd-cl/9ddc9e9a-e98a-4049-b5e8-59f670e3f077/779fb0f9-dcb6-4451-b270-81f01bb7d326","volume":{"uuid":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f5efdbfb-8fe7-3dfd-bc07-5088222d8b84","id":52,"poolType":"RBD","host":"ceph-mon.cloud.bstelecom.ru","path":"rbd-cl","port":6789,"url":"RBD://ceph-mon.cloud.bstelecom.ru/rbd-cl/?ROLE=Primary&STOREUUID=f5efdbfb-8fe7-3dfd-bc07-5088222d8b84"}},"name":"ROOT-105","size":472312242,"path":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeId":382,"vmName":"i-7-105-VM","accountId":7,"format":"RAW","provisioningType":"THIN","id":382,"deviceId":0,"hypervisorType":"KVM"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f5efdbfb-8fe7-3dfd-bc07-5088222d8b84","id":52,"poolType":"RBD","host":"ceph-mon.cloud.bstelecom.ru","path":"rbd-cl","port":6789,"url":"RBD://ceph-mon.cloud.bstelecom.ru/rbd-cl/?ROLE=Primary&STOREUUID=f5efdbfb-8fe7-3dfd-bc07-5088222d8b84"}},"vmName":"i-7-105-VM","name":"test","hypervisorType":"KVM","id":19,"quiescevm":false,"physicalSize":0}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/382","volume":{"uuid":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f5efdbfb-8fe7-3dfd-bc07-5088222d8b84","id":52,"poolType":"RBD","host":"ceph-mon.cloud.bstelecom.ru","path":"rbd-cl","port":6789,"url":"RBD://ceph-mon.cloud.bstelecom.ru/rbd-cl/?ROLE=Primary&STOREUUID=f5efdbfb-8fe7-3dfd-bc07-5088222d8b84"}},"name":"ROOT-105","size":472312242,"path":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeId":382,"vmName":"i-7-105-VM","accountId":7,"format":"RAW","provisioningType":"THIN","id":382,"deviceId":0,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.SwiftTO":{"id":145,"url":"http://s3.cloud.bstelecom.ru/auth/v1.0/","account":"cloudstack","userName":"swift","key":"VPGpdk5ppNgCepV58NNa1E1x4j5RDP9xyIxB7XMB"}},"vmName":"i-7-105-VM","name":"test","hypervisorType":"KVM","id":19,"quiescevm":false,"physicalSize":0}},"cacheTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/382","volume":{"uuid":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f5efdbfb-8fe7-3dfd-bc07-5088222d8b84","id":52,"poolType":"RBD","host":"ceph-mon.cloud.bstelecom.ru","path":"rbd-cl","port":6789,"url":"RBD://ceph-mon.cloud.bstelecom.ru/rbd-cl/?ROLE=Primary&STOREUUID=f5efdbfb-8fe7-3dfd-bc07-5088222d8b84"}},"name":"ROOT-105","size":472312242,"path":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeId":382,"vmName":"i-7-105-VM","accountId":7,"format":"RAW","provisioningType":"THIN","id":382,"deviceId":0,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://srv-nfs.cloud.bstelecom.ru/nfs","_role":"ImageCache"}},"vmName":"i-7-105-VM","name":"test","hypervisorType":"KVM","id":19,"quiescevm":false,"physicalSize":0}},"executeInSequence":true,"options":{"fullSnapshot":"true"},"options2":{},"wait":21600}}] } 2016-02-20 10:25:50,102 DEBUG [c.c.a.ApiServlet] (catalina-exec-9:ctx-a8490da0 ctx-6373cd53) (logid:6886e205) ===END=== 10.30.10.41 -- GET command=queryAsyncJobResult&jobId=a5032eea-7fdc-4f79-a613-1a7822dfabaa&response=json&_=1455953150153 2016-02-20 10:26:47,877 DEBUG [c.c.a.t.Request] (AgentManager-Handler-10:null) (logid:) Seq 121-7199004004351743165: Processing: { Ans: , MgmtId: 90212275930085, via: 121, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Unsupported protocol","wait":0}}] } 2016-02-20 10:26:47,877 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-10:null) (logid:) Seq 121-7199004004351743165: No more commands found 2016-02-20 10:26:47,877 DEBUG [c.c.a.t.Request] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Seq 121-7199004004351743165: Received: { Ans: , MgmtId: 90212275930085, via: 121(srv-kvm01.cloud.bstelecom.ru), Ver: v1, Flags: 110, { CopyCmdAnswer } } 2016-02-20 10:26:47,901 DEBUG [c.c.h.o.r.Ovm3HypervisorGuru] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) getCommandHostDelegation: class org.apache.cloudstack.storage.command.DeleteCommand 2016-02-20 10:26:47,901 DEBUG [c.c.h.XenServerGuru] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) getCommandHostDelegation: class org.apache.cloudstack.storage.command.DeleteCommand 2016-02-20 10:26:47,907 DEBUG [c.c.a.t.Request] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Seq 139-9216335162437012053: Sending { Cmd , MgmtId: 90212275930085, via: 139(s-103-VM), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.DeleteCommand":{"data":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/7/382","volume":{"uuid":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f5efdbfb-8fe7-3dfd-bc07-5088222d8b84","id":52,"poolType":"RBD","host":"ceph-mon.cloud.bstelecom.ru","path":"rbd-cl","port":6789,"url":"RBD://ceph-mon.cloud.bstelecom.ru/rbd-cl/?ROLE=Primary&STOREUUID=f5efdbfb-8fe7-3dfd-bc07-5088222d8b84"}},"name":"ROOT-105","size":472312242,"path":"9ddc9e9a-e98a-4049-b5e8-59f670e3f077","volumeId":382,"vmName":"i-7-105-VM","accountId":7,"format":"RAW","provisioningType":"THIN","id":382,"deviceId":0,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://srv-nfs.cloud.bstelecom.ru/nfs","_role":"ImageCache"}},"vmName":"i-7-105-VM","name":"test","hypervisorType":"KVM","id":19,"quiescevm":false,"physicalSize":0}},"wait":0}}] } 2016-02-20 10:26:47,936 DEBUG [c.c.a.ApiServlet] (catalina-exec-1:ctx-174adeaa) (logid:52a3528c) ===START=== 10.30.10.41 -- GET command=queryAsyncJobResult&jobId=a5032eea-7fdc-4f79-a613-1a7822dfabaa&response=json&_=1455953208093 2016-02-20 10:26:47,996 DEBUG [c.c.u.AccountManagerImpl] (catalina-exec-1:ctx-174adeaa ctx-1cbaa0e4) (logid:52a3528c) Access to Acct[69296515-f19c-407c-859e-42f4599e4653-ykarpel] granted to Acct[69296515-f19c-407c-859e-42f4599e4653-ykarpel] by DomainChecker 2016-02-20 10:26:48,036 DEBUG [c.c.a.ApiServlet] (catalina-exec-1:ctx-174adeaa ctx-1cbaa0e4) (logid:52a3528c) ===END=== 10.30.10.41 -- GET command=queryAsyncJobResult&jobId=a5032eea-7fdc-4f79-a613-1a7822dfabaa&response=json&_=1455953208093 2016-02-20 10:26:48,076 DEBUG [c.c.a.t.Request] (AgentManager-Handler-1:null) (logid:) Seq 139-9216335162437012053: Processing: { Ans: , MgmtId: 90212275930085, via: 139, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":true,"wait":0}}] } 2016-02-20 10:26:48,076 DEBUG [c.c.a.t.Request] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Seq 139-9216335162437012053: Received: { Ans: , MgmtId: 90212275930085, via: 139(s-103-VM), Ver: v1, Flags: 10, { Answer } } 2016-02-20 10:26:48,187 DEBUG [c.c.s.s.SnapshotManagerImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Failed to create snapshot com.cloud.utils.exception.CloudRuntimeException: Unsupported protocol at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:285) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:155) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:322) at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1007) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy189.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2091) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2883) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2891) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy194.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) 2016-02-20 10:26:48,243 DEBUG [o.a.c.s.v.VolumeServiceImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Take snapshot: 382 failed com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1023) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy189.takeSnapshot(Unknown Source) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1591) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2091) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2883) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2891) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy194.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.cloud.utils.exception.CloudRuntimeException: Unsupported protocol at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:285) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:155) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:322) ... 45 more 2016-02-20 10:26:48,246 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Invocation exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to take snapshot 2016-02-20 10:26:48,247 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731 ctx-5ae19c97) (logid:a5032eea) Rethrow exception com.cloud.utils.exception.CloudRuntimeException: Failed to take snapshot 2016-02-20 10:26:48,247 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Done with run of VM work job: com.cloud.vm.VmWorkTakeVolumeSnapshot for VM 105, job origin: 2728 2016-02-20 10:26:48,247 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Unable to complete AsyncJobVO {id:2731, userId: 10, accountId: 7, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkTakeVolumeSnapshot, cmdInfo: rO0ABXNyACVjb20uY2xvdWQudm0uVm1Xb3JrVGFrZVZvbHVtZVNuYXBzaG90BL5gG4Li1c8CAARaAAlxdWllc2NlVm1MAAhwb2xpY3lJZHQAEExqYXZhL2xhbmcvTG9uZztMAApzbmFwc2hvdElkcQB-AAFMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAcAAAAAAAAACgAAAAAAAABpdAAUVm9sdW1lQXBpU2VydmljZUltcGwAc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAAAc3EAfgAGAAAAAAAAABNzcQB-AAYAAAAAAAABfg, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 90212275930085, completeMsid: null, lastUpdated: null, lastPolled: null, created: Sat Feb 20 10:25:47 MSK 2016}, job origin:2728 com.cloud.utils.exception.CloudRuntimeException: Failed to take snapshot at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1594) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2091) at com.cloud.storage.VolumeApiServiceImpl.orchestrateTakeVolumeSnapshot(VolumeApiServiceImpl.java:2883) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2891) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at com.sun.proxy.$Proxy194.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:554) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:502) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1023) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ... 42 more Caused by: com.cloud.utils.exception.CloudRuntimeException: Unsupported protocol at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:285) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:155) at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:322) ... 45 more 2016-02-20 10:26:48,269 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Complete async job-2731, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbHMuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXwZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cgATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMamF2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbGFuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVHJhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdXRpbC9MaXN0O3hwc3EAfgAAc3EAfgAAcQB-AAp0ABRVbnN1cHBvcnRlZCBwcm90b2NvbHVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAAMHNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAFTAAIZmlsZU5hbWVxAH4ABUwACm1ldGhvZE5hbWVxAH4ABXhwAAABHXQAOm9yZy5hcGFjaGUuY2xvdWRzdGFjay5zdG9yYWdlLnNuYXBzaG90LlNuYXBzaG90U2VydmljZUltcGx0ABhTbmFwc2hvdFNlcnZpY2VJbXBsLmphdmF0AA5iYWNrdXBTbmFwc2hvdHNxAH4ADgAAAJt0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS5zbmFwc2hvdC5YZW5zZXJ2ZXJTbmFwc2hvdFN0cmF0ZWd5dAAeWGVuc2VydmVyU25hcHNob3RTdHJhdGVneS5qYXZhcQB-ABJzcQB-AA4AAAFCcQB-ABRxAH4AFXQADHRha2VTbmFwc2hvdHNxAH4ADgAAA-90AC5jb20uY2xvdWQuc3RvcmFnZS5zbmFwc2hvdC5TbmFwc2hvdE1hbmFnZXJJbXBsdAAYU25hcHNob3RNYW5hZ2VySW1wbC5qYXZhcQB-ABdzcQB-AA7____-dAAkc3VuLnJlZmxlY3QuTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsdAAdTmF0aXZlTWV0aG9kQWNjZXNzb3JJbXBsLmphdmF0AAdpbnZva2Uwc3EAfgAOAAAAPnEAfgAccQB-AB10AAZpbnZva2VzcQB-AA4AAAArdAAoc3VuLnJlZmxlY3QuRGVsZWdhdGluZ01ldGhvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcGwuamF2YXEAfgAgc3EAfgAOAAAB8XQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldGhvZHQAC01ldGhvZC5qYXZhcQB-ACBzcQB-AA4AAAE9dAAob3JnLnNwcmluZ2ZyYW1ld29yay5hb3Auc3VwcG9ydC5Bb3BVdGlsc3QADUFvcFV0aWxzLmphdmF0AB5pbnZva2VKb2lucG9pbnRVc2luZ1JlZmxlY3Rpb25zcQB-AA4AAAC3dAA8b3JnLnNwcmluZ2ZyYW1ld29yay5hb3AuZnJhbWV3b3JrLlJlZmxlY3RpdmVNZXRob2RJbnZvY2F0aW9udAAfUmVmbGVjdGl2ZU1ldGhvZEludm9jYXRpb24uamF2YXQAD2ludm9rZUpvaW5wb2ludHNxAH4ADgAAAJZxAH4ALHEAfgAtdAAHcHJvY2VlZHNxAH4ADgAAAFt0AD9vcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5pbnRlcmNlcHRvci5FeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3J0ACBFeHBvc2VJbnZvY2F0aW9uSW50ZXJjZXB0b3IuamF2YXEAfgAgc3EAfgAOAAAArHEAfgAscQB-AC1xAH4AMHNxAH4ADgAAAMx0ADRvcmcuc3ByaW5nZnJhbWV3b3JrLmFvcC5mcmFtZXdvcmsuSmRrRHluYW1pY0FvcFByb3h5dAAXSmRrRHluYW1pY0FvcFByb3h5LmphdmFxAH4AIHNxAH4ADv____90ABdjb20uc3VuLnByb3h5LiRQcm94eTE4OXBxAH4AF3NxAH4ADgAABjd0ADZvcmcuYXBhY2hlLmNsb3Vkc3RhY2suc3RvcmFnZS52b2x1bWUuVm9sdW1lU2VydmljZUltcGx0ABZWb2x1bWVTZXJ2aWNlSW1wbC5qYXZhcQB-ABdzcQB-AA4AAAgrdAAmY29tLmNsb3VkLnN0b3JhZ2UuVm9sdW1lQXBpU2VydmljZUltcGx0ABlWb2x1bWVBcGlTZXJ2aWNlSW1wbC5qYXZhdAAdb3JjaGVzdHJhdGVUYWtlVm9sdW1lU25hcHNob3RzcQB-AA4AAAtDcQB-AD5xAH4AP3EAfgBAc3EAfgAO_____nEAfgAccQB-AB1xAH4AHnNxAH4ADgAAAD5xAH4AHHEAfgAdcQB-ACBzcQB-AA4AAAArcQB-ACJxAH4AI3EAfgAgc3EAfgAOAAAB8XEAfgAlcQB-ACZxAH4AIHNxAH4ADgAAAGt0ACJjb20uY2xvdWQudm0uVm1Xb3JrSm9iSGFuZGxlclByb3h5dAAaVm1Xb3JrSm9iSGFuZGxlclByb3h5LmphdmF0AA9oYW5kbGVWbVdvcmtKb2JzcQB-AA4AAAtLcQB-AD5xAH4AP3EAfgBJc3EAfgAO_____nEAfgAccQB-AB1xAH4AHnNxAH4ADgAAAD5xAH4AHHEAfgAdcQB-ACBzcQB-AA4AAAArcQB-ACJxAH4AI3EAfgAgc3EAfgAOAAAB8XEAfgAlcQB-ACZxAH4AIHNxAH4ADgAAAT1xAH4AKHEAfgApcQB-ACpzcQB-AA4AAAC3cQB-ACxxAH4ALXEAfgAuc3EAfgAOAAAAlnEAfgAscQB-AC1xAH4AMHNxAH4ADgAAAFtxAH4AMnEAfgAzcQB-ACBzcQB-AA4AAACscQB-ACxxAH4ALXEAfgAwc3EAfgAOAAAAzHEAfgA2cQB-ADdxAH4AIHNxAH4ADv____90ABdjb20uc3VuLnByb3h5LiRQcm94eTE5NHBxAH4ASXNxAH4ADgAAAGZ0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRGlzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYnNxAH4ADgAAAip0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLmpvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbmFnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQB-AA4AAAAxdAA-b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AANydW5zcQB-AA4AAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYXVsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfgAOAAAAZ3QAQG9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZENvbnRleHRxAH4AZXQAD2NhbGxXaXRoQ29udGV4dHNxAH4ADgAAADVxAH4AaHEAfgBldAAOcnVuV2l0aENvbnRleHRzcQB-AA4AAAAudAA8b3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udGV4dFJ1bm5hYmxlcQB-AGFxAH4AYnNxAH4ADgAAAfZxAH4AXHEAfgBdcQB-AGJzcQB-AA4AAAH_dAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-AGZzcQB-AA4AAAEKdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dHVyZVRhc2suamF2YXEAfgBic3EAfgAOAAAEdnQAJ2phdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci5qYXZhdAAJcnVuV29ya2Vyc3EAfgAOAAACaXQALmphdmEudXRpbC5jb25jdXJyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4Ad3EAfgBic3EAfgAOAAAC6XQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgBic3IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0_A8lMbXsjhACAAFMAARsaXN0cQB-AAd4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4Ag3gAABCadwgAAAAAAAAAAHh0ABlGYWlsZWQgdG8gY3JlYXRlIHNuYXBzaG90dXEAfgAMAAAALXNxAH4ADgAAA_9xAH4AGXEAfgAacQB-ABdzcQB-AA7____-cQB-ABxxAH4AHXEAfgAec3EAfgAOAAAAPnEAfgAccQB-AB1xAH4AIHNxAH4ADgAAACtxAH4AInEAfgAjcQB-ACBzcQB-AA4AAAHxcQB-ACVxAH4AJnEAfgAgc3EAfgAOAAABPXEAfgAocQB-AClxAH4AKnNxAH4ADgAAALdxAH4ALHEAfgAtcQB-AC5zcQB-AA4AAACWcQB-ACxxAH4ALXEAfgAwc3EAfgAOAAAAW3EAfgAycQB-ADNxAH4AIHNxAH4ADgAAAKxxAH4ALHEAfgAtcQB-ADBzcQB-AA4AAADMcQB-ADZxAH4AN3EAfgAgc3EAfgAO_____3EAfgA5cHEAfgAXc3EAfgAOAAAGN3EAfgA7cQB-ADxxAH4AF3NxAH4ADgAACCtxAH4APnEAfgA_cQB-AEBzcQB-AA4AAAtDcQB-AD5xAH4AP3EAfgBAc3EAfgAO_____nEAfgAccQB-AB1xAH4AHnNxAH4ADgAAAD5xAH4AHHEAfgAdcQB-ACBzcQB-AA4AAAArcQB-ACJxAH4AI3EAfgAgc3EAfgAOAAAB8XEAfgAlcQB-ACZxAH4AIHNxAH4ADgAAAGtxAH4AR3EAfgBIcQB-AElzcQB-AA4AAAtLcQB-AD5xAH4AP3EAfgBJc3EAfgAO_____nEAfgAccQB-AB1xAH4AHnNxAH4ADgAAAD5xAH4AHHEAfgAdcQB-ACBzcQB-AA4AAAArcQB-ACJxAH4AI3EAfgAgc3EAfgAOAAAB8XEAfgAlcQB-ACZxAH4AIHNxAH4ADgAAAT1xAH4AKHEAfgApcQB-ACpzcQB-AA4AAAC3cQB-ACxxAH4ALXEAfgAuc3EAfgAOAAAAlnEAfgAscQB-AC1xAH4AMHNxAH4ADgAAAFtxAH4AMnEAfgAzcQB-ACBzcQB-AA4AAACscQB-ACxxAH4ALXEAfgAwc3EAfgAOAAAAzHEAfgA2cQB-ADdxAH4AIHNxAH4ADv____9xAH4AVnBxAH4ASXNxAH4ADgAAAGZxAH4AWHEAfgBZcQB-AFpzcQB-AA4AAAIqcQB-AFxxAH4AXXEAfgBec3EAfgAOAAAAMXEAfgBgcQB-AGFxAH4AYnNxAH4ADgAAADhxAH4AZHEAfgBlcQB-AGZzcQB-AA4AAABncQB-AGhxAH4AZXEAfgBpc3EAfgAOAAAANXEAfgBocQB-AGVxAH4Aa3NxAH4ADgAAAC5xAH4AbXEAfgBhcQB-AGJzcQB-AA4AAAH2cQB-AFxxAH4AXXEAfgBic3EAfgAOAAAB_3EAfgBwcQB-AHFxAH4AZnNxAH4ADgAAAQpxAH4Ac3EAfgB0cQB-AGJzcQB-AA4AAAR2cQB-AHZxAH4Ad3EAfgB4c3EAfgAOAAACaXEAfgB6cQB-AHdxAH4AYnNxAH4ADgAAAulxAH4AfHEAfgB9cQB-AGJxAH4AgXgAABCadwgAAAAAAAAAAHh0ABdGYWlsZWQgdG8gdGFrZSBzbmFwc2hvdHVxAH4ADAAAACFzcQB-AA4AAAY6cQB-ADtxAH4APHEAfgAXc3EAfgAOAAAIK3EAfgA-cQB-AD9xAH4AQHNxAH4ADgAAC0NxAH4APnEAfgA_cQB-AEBzcQB-AA7____-cQB-ABxxAH4AHXEAfgAec3EAfgAOAAAAPnEAfgAccQB-AB1xAH4AIHNxAH4ADgAAACtxAH4AInEAfgAjcQB-ACBzcQB-AA4AAAHxcQB-ACVxAH4AJnEAfgAgc3EAfgAOAAAAa3EAfgBHcQB-AEhxAH4ASXNxAH4ADgAAC0txAH4APnEAfgA_cQB-AElzcQB-AA7____-cQB-ABxxAH4AHXEAfgAec3EAfgAOAAAAPnEAfgAccQB-AB1xAH4AIHNxAH4ADgAAACtxAH4AInEAfgAjcQB-ACBzcQB-AA4AAAHxcQB-ACVxAH4AJnEAfgAgc3EAfgAOAAABPXEAfgAocQB-AClxAH4AKnNxAH4ADgAAALdxAH4ALHEAfgAtcQB-AC5zcQB-AA4AAACWcQB-ACxxAH4ALXEAfgAwc3EAfgAOAAAAW3EAfgAycQB-ADNxAH4AIHNxAH4ADgAAAKxxAH4ALHEAfgAtcQB-ADBzcQB-AA4AAADMcQB-ADZxAH4AN3EAfgAgc3EAfgAO_____3EAfgBWcHEAfgBJc3EAfgAOAAAAZnEAfgBYcQB-AFlxAH4AWnNxAH4ADgAAAipxAH4AXHEAfgBdcQB-AF5zcQB-AA4AAAAxcQB-AGBxAH4AYXEAfgBic3EAfgAOAAAAOHEAfgBkcQB-AGVxAH4AZnNxAH4ADgAAAGdxAH4AaHEAfgBlcQB-AGlzcQB-AA4AAAA1cQB-AGhxAH4AZXEAfgBrc3EAfgAOAAAALnEAfgBtcQB-AGFxAH4AYnNxAH4ADgAAAfZxAH4AXHEAfgBdcQB-AGJzcQB-AA4AAAH_cQB-AHBxAH4AcXEAfgBmc3EAfgAOAAABCnEAfgBzcQB-AHRxAH4AYnNxAH4ADgAABHZxAH4AdnEAfgB3cQB-AHhzcQB-AA4AAAJpcQB-AHpxAH4Ad3EAfgBic3EAfgAOAAAC6XEAfgB8cQB-AH1xAH4AYnEAfgCBeAAAEJp3CAAAAAAAAAAAeA 2016-02-20 10:26:48,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Publish async job-2731 complete on message bus 2016-02-20 10:26:48,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Wake up jobs related to job-2731 2016-02-20 10:26:48,276 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Update db status for job-2731 2016-02-20 10:26:48,282 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Wake up jobs joined with job-2731 and disjoin all subjobs created from job- 2731 2016-02-20 10:26:48,299 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Done executing com.cloud.vm.VmWorkTakeVolumeSnapshot for job-2731 2016-02-20 10:26:48,307 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-2:ctx-6fb5a754 job-2728/job-2731) (logid:a5032eea) Remove job-2731 from job monitoring 2016-02-20 10:26:48,365 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-22ae7862 job-2728) (logid:a5032eea) Complete async job-2728, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 382"} 2016-02-20 10:26:48,370 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-22ae7862 job-2728) (logid:a5032eea) Publish async job-2728 complete on message bus 2016-02-20 10:26:48,370 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-22ae7862 job-2728) (logid:a5032eea) Wake up jobs related to job-2728 2016-02-20 10:26:48,370 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-22ae7862 job-2728) (logid:a5032eea) Update db status for job-2728 2016-02-20 10:26:48,375 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-22ae7862 job-2728) (logid:a5032eea) Wake up jobs joined with job-2728 and disjoin all subjobs created from job- 2728 2016-02-20 10:26:48,383 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-2:ctx-22ae7862 job-2728) (logid:a5032eea) Done executing org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd for job-2728 2016-02-20 10:26:48,383 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-2:ctx-22ae7862 job-2728) (logid:a5032eea) Remove job-2728 from job monitoring