[CLOUDSTACK-5357] Xenserver - Failed to create snapshot due to "unable to destroy task(com.xe nsource.xenapi.Task@67d312d6) on host(23af93a0-93ff-40cb-ba11-a11d1b884d37)" when secondary store was unavaiable for 1 and 1/2 hours and then brought up..

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Won't Fix
• Affects Version/s: 4.3.0
• Component/s: Management Server
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: Build from 4.3
• Assignee: Anthony Xu
• Reporter: Sangeetha Hariharan
• Created: 04/Dec/13 00:57
• Updated: 15/Oct/14 22:25
• Resolved: 15/Oct/14 22:25

Description

Xenserver - Failed to create snapshot due to "unable to destroy task(com.xe nsource.xenapi.Task@67d312d6) on host(23af93a0-93ff-40cb-ba11-a11d1b884d37)" when secondary store was unavaiable for 1 and 1/2 hours and then brought up.

Set up:

Advanced Zone with 2 Xenserver 6.2 hosts:

Steps to reproduce the problem:

1. Deploy 5 Vms in each of the hosts with 10 GB ROOT volume size , so we start with 10 Vms.
2. Start concurrent hourly snapshots for ROOT volumes of all the Vms.
3. Shutdown the Secondary storage server when the snapshots are in the progress.
4. Bring the Secondary storage server up after 1and 1/2 hour.

After the secondary store was brought up , we see the snapshot process continue to be active ( I see the copy command that was halted when the secondary store was down , starts progressing when server is drought up online).

I also see that the snapshot process has been completed on the secondary store. But management server throws the following exception and leaves behind the snapshot in "CreatedOnPrimary" state in DB.

2013-12-03 19:24:36,296 WARN [c.c.h.x.r.CitrixResourceBase] (DirectAgent-252:ctx-8ca50e48) unable to destroy task(com.xe
nsource.xenapi.Task@67d312d6) on host(23af93a0-93ff-40cb-ba11-a11d1b884d37) due to
You gave an invalid object reference. The object may have recently been deleted. The class parameter gives the type of
reference given, and the handle parameter echoes the bad value given.
at com.xensource.xenapi.Types.checkResponse(Types.java:209)
at com.xensource.xenapi.Connection.dispatch(Connection.java:368)
at com.cloud.hypervisor.xen.resource.XenServerConnectionPool$XenServerConnection.dispatch(XenServerConnectionPool
.java:909)
at com.xensource.xenapi.Task.destroy(Task.java:616)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.cloudVDIcopy(CitrixResourceBase.java:3920)
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.backupSnapshot(XenServerStorageProcessor.java:1268
)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:
90)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHan
dlerBase.java:50)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:613)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:10
3)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.ja
va:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2013-12-03 19:24:36,297 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-252:ctx-8ca50e48) Host 10.223.59.67 OpaqueRef:50d81c75-f239-1746-ae06-c6228e788104: Removing SR
2013-12-03 19:24:36,305 DEBUG [c.c.h.x.r.CitrixResourceBase] (DirectAgent-252:ctx-8ca50e48) Host 10.223.59.67 OpaqueRef:e2bcc8f5-a04a-3f5a-c234-85beb190aa02: Unplugging pbd

[root@Rack3Host5 6]# ls -lth
total 1.3G
rw-rr-. 1 root root 6.7G Dec 3 23:24 1f4ec466-688f-4a53-bfcb-e5f2da3bccae.vhd
[root@Rack3Host5 6]#

Issue Links

• relates to: CLOUDSTACK-5356 Xenserver - Failed to create snapshot when secondary store was made unavaibale for about 1.5 hour leaving behind snapshot in " CreatedOnPrimary" state. The subsequent scheduled snapshot also failed
• relates to: CLOUDSTACK-5469 Snapshot creation fails with following exception - "Failed to backup snapshot: qemu-img: Could not delete snapshot '89eced14-9121-44a7-bb97-26b567795726': -2 (No such file or directory)"