[CLOUDSTACK-5800] While creating a VM from template (which is created based on existing newly created vm) getting error as Unable to deploy vm base on template due to of insufficient server capicity

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Cannot Reproduce
• Affects Version/s: 4.3.0
• Fix Version/s: 4.4.0, 4.5.0
• Component/s: Template, XenServer
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: CENTOS 6.4,WIN2012,ACS 4.3.0,
• Assignee: Anthony Xu
• Reporter: rashid
• Created: 06/Jan/14 13:58
• Updated: 19/Nov/14 00:35
• Resolved: 19/Nov/14 00:35

Description

While creating a vm from template

INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-5:ctx-c259052d) Add job-46 into job monitoring
INFO [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-5:ctx-c259052d ctx-506ea49e) lock is acquired for VMTemplateStoragePool 7
WARN [o.a.c.alerts] (CapacityChecker:ctx-374c70b5) alertType:: 24 // dataCenterId:: 1 // podId:: null // clusterId:: null // message:: System Alert: Number of unallocated shared network IPs is low in availability zone NEWZONE
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-2:ctx-32c31ce6) Task (job-45) has been pending for 104 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-2:ctx-32c31ce6) Task (job-46) has been pending for 103 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-2:ctx-92c2ed53) Task (job-45) has been pending for 164 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-2:ctx-92c2ed53) Task (job-46) has been pending for 163 seconds
WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-4:ctx-f87acd3c) destoryVDIbyNameLabel failed due to there are 0 VDIs with name cloud-377c12f4-a8e9-491a-8cba-34b9532455f8
WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-4:ctx-f87acd3c) failed to set hidden to 0 /var/run/sr-mount/ab820f54-badd-db21-f758-0abcb47ea4d7/b0246f72-9bda-4f29-981f-548f4b7a9f6c.vhd
WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-4:ctx-f87acd3c) Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: failed to set hidden to 0 /var/run/sr-mount/ab820f54-badd-db21-f758-0abcb47ea4d7/b0246f72-9bda-4f29-981f-548f4b7a9f6c.vhd
com.cloud.utils.exception.CloudRuntimeException: failed to set hidden to 0 /var/run/sr-mount/ab820f54-badd-db21-f758-0abcb47ea4d7/b0246f72-9bda-4f29-981f-548f4b7a9f6c.vhd
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:858)
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:928)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:75)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:609)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-5:ctx-c259052d ctx-506ea49e) releasing lock for VMTemplateStoragePool 7
WARN [c.c.u.d.Merovingian2] (Job-Executor-5:ctx-c259052d ctx-506ea49e) Was unable to find lock for the key template_spool_ref7 and thread id 542318778
INFO [c.c.v.VirtualMachineManagerImpl] (Job-Executor-5:ctx-c259052d ctx-506ea49e) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:3] is unreachable: Unable to create Vol[12|vm=9|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: failed to set hidden to 0 /var/run/sr-mount/ab820f54-badd-db21-f758-0abcb47ea4d7/b0246f72-9bda-4f29-981f-548f4b7a9f6c.vhd
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1160)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1202)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:943)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4736)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4855)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:524)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-15:ctx-1d499a69) VM does not exist on XenServerf6a356da-010b-49a5-be76-71e556f602e3
ERROR [c.c.v.VmWorkJobHandlerProxy] (Job-Executor-5:ctx-c259052d ctx-506ea49e) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|win2012vmtemplate]Scope=interface com.cloud.dc.DataCenter; id=1
ERROR [c.c.v.VmWorkJobDispatcher] (Job-Executor-5:ctx-c259052d ctx-506ea49e) Unable to complete AsyncJobVO

{id:46, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIACkoABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAACXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXBzcgAOamF2YS5sYW5nLkxvbmc7i-SQzI8j3wIAAUoABXZhbHVleHIAEGphdmEubGFuZy5OdW1iZXKGrJUdC5TgiwIAAHhwAAAAAAAAAAFxAH4ACnBwcQB-AApwc3IAEWphdmEudXRpbC5IYXNoTWFwBQfawcMWYNEDAAJGAApsb2FkRmFjdG9ySQAJdGhyZXNob2xkeHA_QAAAAAAADHcIAAAAEAAAAAF0AApWbVBhc3N3b3JkdAAcck8wQUJYUUFEbk5oZG1Wa1gzQmhjM04zYjNKa3hw, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052565034, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan 03 06:14:34 EST 2014}

com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|win2012vmtemplate]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:902)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4736)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4855)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:524)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO [c.c.v.VmWorkJobDispatcher] (Job-Executor-5:ctx-c259052d ctx-506ea49e) Serialize exception object into json: com.cloud.exception.InsufficientServerCapacityException/{"affinityGroupsApplied":false,"id":1,"scope":"com.cloud.dc.DataCenter","idList":[],"csErrorCode":4335,"detailMessage":"Unable to create a deployment for VM[User|win2012vmtemplate]","cause":

{"class":"com.cloud.exception.InsufficientServerCapacityException","msg":"Unable to create a deployment for VM[User|win2012vmtemplate]"}

,"stackTrace":[

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"orchestrateStart","fileName":"VirtualMachineManagerImpl.java","lineNumber":902}

,

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"orchestrateStart","fileName":"VirtualMachineManagerImpl.java","lineNumber":4736}

,

{"declaringClass":"sun.reflect.NativeMethodAccessorImpl","methodName":"invoke0","fileName":"NativeMethodAccessorImpl.java","lineNumber":-2}

,

{"declaringClass":"sun.reflect.NativeMethodAccessorImpl","methodName":"invoke","fileName":"NativeMethodAccessorImpl.java","lineNumber":57}

,

{"declaringClass":"sun.reflect.DelegatingMethodAccessorImpl","methodName":"invoke","fileName":"DelegatingMethodAccessorImpl.java","lineNumber":43}

,

{"declaringClass":"java.lang.reflect.Method","methodName":"invoke","fileName":"Method.java","lineNumber":606}

,

{"declaringClass":"com.cloud.vm.VmWorkJobHandlerProxy","methodName":"handleVmWorkJob","fileName":"VmWorkJobHandlerProxy.java","lineNumber":107}

,

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"handleVmWorkJob","fileName":"VirtualMachineManagerImpl.java","lineNumber":4855}

,

{"declaringClass":"com.cloud.vm.VmWorkJobDispatcher","methodName":"runJob","fileName":"VmWorkJobDispatcher.java","lineNumber":99}

,

{"declaringClass":"org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5","methodName":"runInContext","fileName":"AsyncJobManagerImpl.java","lineNumber":524}

,

{"declaringClass":"org.apache.cloudstack.managed.context.ManagedContextRunnable$1","methodName":"run","fileName":"ManagedContextRunnable.java","lineNumber":49}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1","methodName":"call","fileName":"DefaultManagedContext.java","lineNumber":56}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext","methodName":"callWithContext","fileName":"DefaultManagedContext.java","lineNumber":103}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext","methodName":"runWithContext","fileName":"DefaultManagedContext.java","lineNumber":53}

,

{"declaringClass":"org.apache.cloudstack.managed.context.ManagedContextRunnable","methodName":"run","fileName":"ManagedContextRunnable.java","lineNumber":46}

,

{"declaringClass":"java.util.concurrent.Executors$RunnableAdapter","methodName":"call","fileName":"Executors.java","lineNumber":471}

,

{"declaringClass":"java.util.concurrent.FutureTask$Sync","methodName":"innerRun","fileName":"FutureTask.java","lineNumber":334}

,

{"declaringClass":"java.util.concurrent.FutureTask","methodName":"run","fileName":"FutureTask.java","lineNumber":166}

,

{"declaringClass":"java.util.concurrent.ThreadPoolExecutor","methodName":"runWorker","fileName":"ThreadPoolExecutor.java","lineNumber":1145}

,

{"declaringClass":"java.util.concurrent.ThreadPoolExecutor$Worker","methodName":"run","fileName":"ThreadPoolExecutor.java","lineNumber":615}

,

{"declaringClass":"java.lang.Thread","methodName":"run","fileName":"Thread.java","lineNumber":724}

],"suppressedExceptions":[]}
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-6:ctx-d9857e73) Add job-45 into job monitoring
WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-6:ctx-d9857e73) job-45 is scheduled for wakeup run, but there is no joining info anymore
ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-6:ctx-d9857e73) Unable to find a wakeup dispatcher from the joined job: AsyncJobVO {id:45, userId: 2, accountId: 2, instanceType: VirtualMachine, instanceId: 9, cmd: org.apache.cloudstack.api.command.user.vm.DeployVMCmd, cmdInfo:

{"sessionkey":"vbJloaDpWO/+kAdgpxWTcOjq7FY\u003d","serviceofferingid":"bfa66236-af6e-45aa-bc2a-297ff69f490c","cmdEventType":"VM.CREATE","ctxUserId":"2","zoneid":"4e19f84d-df6b-4c1d-b71a-72df0c2c6a5a","httpmethod":"GET","templateid":"ad59295b-1bcd-4fa4-8043-435903bd79ec","securitygroupids":"156467b8-743f-11e3-8a5d-005056bc4e2a","id":"9","response":"json","hypervisor":"XenServer","name":"win2012vmtemplate","_":"1388747674027","ctxAccountId":"2","diskofferingid":"0092282f-3495-4827-8e08-f85b565b807c","ctxStartEventId":"84","displayname":"win2012vmtemplate"}

, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052565034, completeMsid: null, lastUpdated: null, lastPolled: Fri Jan 03 06:17:25 EST 2014, created: Fri Jan 03 06:14:34 EST 2014}
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-6:ctx-d9857e73) Remove job-45 from job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-7:ctx-d6c75107) Add job-47 into job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-5:ctx-c259052d) Remove job-46 from job monitoring
INFO [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-7:ctx-d6c75107 ctx-506ea49e) lock is acquired for VMTemplateStoragePool 8
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-2:ctx-80671640) Task (job-45) has been pending for 224 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-2:ctx-95453627) Task (job-45) has been pending for 284 seconds
WARN [o.a.c.f.j.i.AsyncJobMonitor] (Timer-2:ctx-95453627) Task (job-47) has been pending for 112 seconds
WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-40:ctx-824b2f1f) destoryVDIbyNameLabel failed due to there are 0 VDIs with name cloud-f66c3289-0c18-4136-a986-1ce96f1deb58
WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-40:ctx-824b2f1f) failed to set hidden to 0 /var/run/sr-mount/bbc32540-c84f-c78b-1399-b52cb5b554c5/ae19b574-b80f-47b8-ae6d-c93f7c743901.vhd
WARN [c.c.h.x.r.XenServerStorageProcessor] (DirectAgent-40:ctx-824b2f1f) Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: failed to set hidden to 0 /var/run/sr-mount/bbc32540-c84f-c78b-1399-b52cb5b554c5/ae19b574-b80f-47b8-ae6d-c93f7c743901.vhd
com.cloud.utils.exception.CloudRuntimeException: failed to set hidden to 0 /var/run/sr-mount/bbc32540-c84f-c78b-1399-b52cb5b554c5/ae19b574-b80f-47b8-ae6d-c93f7c743901.vhd
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copy_vhd_from_secondarystorage(XenServerStorageProcessor.java:858)
at com.cloud.hypervisor.xen.resource.XenServerStorageProcessor.copyTemplateToPrimaryStorage(XenServerStorageProcessor.java:928)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.execute(StorageSubsystemCommandHandlerBase.java:75)
at com.cloud.storage.resource.StorageSubsystemCommandHandlerBase.handleStorageCommands(StorageSubsystemCommandHandlerBase.java:50)
at com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:609)
at com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59)
at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:106)
at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:216)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO [o.a.c.s.v.VolumeServiceImpl] (Job-Executor-7:ctx-d6c75107 ctx-506ea49e) releasing lock for VMTemplateStoragePool 8
WARN [c.c.u.d.Merovingian2] (Job-Executor-7:ctx-d6c75107 ctx-506ea49e) Was unable to find lock for the key template_spool_ref8 and thread id 615450567
INFO [c.c.v.VirtualMachineManagerImpl] (Job-Executor-7:ctx-d6c75107 ctx-506ea49e) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:1] is unreachable: Unable to create Vol[12|vm=9|ROOT]:Catch Exception com.cloud.utils.exception.CloudRuntimeException for template + due to com.cloud.utils.exception.CloudRuntimeException: failed to set hidden to 0 /var/run/sr-mount/bbc32540-c84f-c78b-1399-b52cb5b554c5/ae19b574-b80f-47b8-ae6d-c93f7c743901.vhd
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.recreateVolume(VolumeOrchestrator.java:1160)
at org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.prepare(VolumeOrchestrator.java:1202)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:943)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4736)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4855)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:524)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO [c.c.h.x.r.CitrixResourceBase] (DirectAgent-23:ctx-dccd8b36) VM does not exist on XenServerf6a356da-010b-49a5-be76-71e556f602e3
ERROR [c.c.v.VmWorkJobHandlerProxy] (Job-Executor-7:ctx-d6c75107 ctx-506ea49e) Invocation exception, caused by: com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|win2012vmtemplate]Scope=interface com.cloud.dc.DataCenter; id=1
ERROR [c.c.v.VmWorkJobDispatcher] (Job-Executor-7:ctx-d6c75107 ctx-506ea49e) Unable to complete AsyncJobVO

{id:47, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkStart, cmdInfo: rO0ABXNyABhjb20uY2xvdWQudm0uVm1Xb3JrU3RhcnR9cMGsvxz73gIACkoABGRjSWRMAAZhdm9pZHN0ADBMY29tL2Nsb3VkL2RlcGxveS9EZXBsb3ltZW50UGxhbm5lciRFeGNsdWRlTGlzdDtMAAljbHVzdGVySWR0ABBMamF2YS9sYW5nL0xvbmc7TAAGaG9zdElkcQB-AAJMAAtqb3VybmFsTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO0wAEXBoeXNpY2FsTmV0d29ya0lkcQB-AAJMAAVwb2RJZHEAfgACTAAGcG9vbElkcQB-AAJMAAlyYXdQYXJhbXN0AA9MamF2YS91dGlsL01hcDtMAA1yZXNlcnZhdGlvbklkcQB-AAN4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1lcQB-AAN4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAACXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAHBwcHBwcHBzcgARamF2YS51dGlsLkhhc2hNYXAFB9rBwxZg0QMAAkYACmxvYWRGYWN0b3JJAAl0aHJlc2hvbGR4cD9AAAAAAAAMdwgAAAAQAAAAAXQAClZtUGFzc3dvcmR0ABxyTzBBQlhRQURuTmhkbVZrWDNCaGMzTjNiM0preHA, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052565034, completeMsid: null, lastUpdated: null, lastPolled: null, created: Fri Jan 03 06:17:26 EST 2014}

com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[User|win2012vmtemplate]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:902)
at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:4736)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4855)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:524)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
INFO [c.c.v.VmWorkJobDispatcher] (Job-Executor-7:ctx-d6c75107 ctx-506ea49e) Serialize exception object into json: com.cloud.exception.InsufficientServerCapacityException/{"affinityGroupsApplied":false,"id":1,"scope":"com.cloud.dc.DataCenter","idList":[],"csErrorCode":4335,"detailMessage":"Unable to create a deployment for VM[User|win2012vmtemplate]","cause":

{"class":"com.cloud.exception.InsufficientServerCapacityException","msg":"Unable to create a deployment for VM[User|win2012vmtemplate]"}

,"stackTrace":[

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"orchestrateStart","fileName":"VirtualMachineManagerImpl.java","lineNumber":902}

,

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"orchestrateStart","fileName":"VirtualMachineManagerImpl.java","lineNumber":4736}

,

{"declaringClass":"sun.reflect.NativeMethodAccessorImpl","methodName":"invoke0","fileName":"NativeMethodAccessorImpl.java","lineNumber":-2}

,

{"declaringClass":"sun.reflect.NativeMethodAccessorImpl","methodName":"invoke","fileName":"NativeMethodAccessorImpl.java","lineNumber":57}

,

{"declaringClass":"sun.reflect.DelegatingMethodAccessorImpl","methodName":"invoke","fileName":"DelegatingMethodAccessorImpl.java","lineNumber":43}

,

{"declaringClass":"java.lang.reflect.Method","methodName":"invoke","fileName":"Method.java","lineNumber":606}

,

{"declaringClass":"com.cloud.vm.VmWorkJobHandlerProxy","methodName":"handleVmWorkJob","fileName":"VmWorkJobHandlerProxy.java","lineNumber":107}

,

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"handleVmWorkJob","fileName":"VirtualMachineManagerImpl.java","lineNumber":4855}

,

{"declaringClass":"com.cloud.vm.VmWorkJobDispatcher","methodName":"runJob","fileName":"VmWorkJobDispatcher.java","lineNumber":99}

,

{"declaringClass":"org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5","methodName":"runInContext","fileName":"AsyncJobManagerImpl.java","lineNumber":524}

,

{"declaringClass":"org.apache.cloudstack.managed.context.ManagedContextRunnable$1","methodName":"run","fileName":"ManagedContextRunnable.java","lineNumber":49}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1","methodName":"call","fileName":"DefaultManagedContext.java","lineNumber":56}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext","methodName":"callWithContext","fileName":"DefaultManagedContext.java","lineNumber":103}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext","methodName":"runWithContext","fileName":"DefaultManagedContext.java","lineNumber":53}

,

{"declaringClass":"org.apache.cloudstack.managed.context.ManagedContextRunnable","methodName":"run","fileName":"ManagedContextRunnable.java","lineNumber":46}

,

{"declaringClass":"java.util.concurrent.Executors$RunnableAdapter","methodName":"call","fileName":"Executors.java","lineNumber":471}

,

{"declaringClass":"java.util.concurrent.FutureTask$Sync","methodName":"innerRun","fileName":"FutureTask.java","lineNumber":334}

,

{"declaringClass":"java.util.concurrent.FutureTask","methodName":"run","fileName":"FutureTask.java","lineNumber":166}

,

{"declaringClass":"java.util.concurrent.ThreadPoolExecutor","methodName":"runWorker","fileName":"ThreadPoolExecutor.java","lineNumber":1145}

,

{"declaringClass":"java.util.concurrent.ThreadPoolExecutor$Worker","methodName":"run","fileName":"ThreadPoolExecutor.java","lineNumber":615}

,

{"declaringClass":"java.lang.Thread","methodName":"run","fileName":"Thread.java","lineNumber":724}

],"suppressedExceptions":[]}
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-8:ctx-351351a4) Add job-45 into job monitoring
WARN [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-8:ctx-351351a4) job-45 is scheduled for wakeup run, but there is no joining info anymore
ERROR [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-8:ctx-351351a4) Unable to find a wakeup dispatcher from the joined job: AsyncJobVO {id:45, userId: 2, accountId: 2, instanceType: VirtualMachine, instanceId: 9, cmd: org.apache.cloudstack.api.command.user.vm.DeployVMCmd, cmdInfo:

{"sessionkey":"vbJloaDpWO/+kAdgpxWTcOjq7FY\u003d","serviceofferingid":"bfa66236-af6e-45aa-bc2a-297ff69f490c","cmdEventType":"VM.CREATE","ctxUserId":"2","zoneid":"4e19f84d-df6b-4c1d-b71a-72df0c2c6a5a","httpmethod":"GET","templateid":"ad59295b-1bcd-4fa4-8043-435903bd79ec","securitygroupids":"156467b8-743f-11e3-8a5d-005056bc4e2a","id":"9","response":"json","hypervisor":"XenServer","name":"win2012vmtemplate","_":"1388747674027","ctxAccountId":"2","diskofferingid":"0092282f-3495-4827-8e08-f85b565b807c","ctxStartEventId":"84","displayname":"win2012vmtemplate"}

, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345052565034, completeMsid: null, lastUpdated: null, lastPolled: Fri Jan 03 06:20:04 EST 2014, created: Fri Jan 03 06:14:34 EST 2014}
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-8:ctx-351351a4) Remove job-45 from job monitoring
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-7:ctx-d6c75107) Remove job-47 from job monitoring
WARN [o.a.c.alerts] (Job-Executor-4:ctx-43a4d668 ctx-506ea49e) alertType:: 8 // dataCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Failed to deploy Vm with Id: 9, on Host with Id: null
ERROR [c.c.a.ApiAsyncJobDispatcher] (Job-Executor-4:ctx-43a4d668) Unexpected exception while executing org.apache.cloudstack.api.command.user.vm.DeployVMCmd
com.cloud.utils.exception.CloudRuntimeException: Unable to serialize: com.cloud.exception.InsufficientServerCapacityException/{"affinityGroupsApplied":false,"id":1,"scope":"com.cloud.dc.DataCenter","idList":[],"csErrorCode":4335,"detailMessage":"Unable to create a deployment for VM[User|win2012vmtemplate]","cause":

{"class":"com.cloud.exception.InsufficientServerCapacityException","msg":"Unable to create a deployment for VM[User|win2012vmtemplate]"}

,"stackTrace":[

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"orchestrateStart","fileName":"VirtualMachineManagerImpl.java","lineNumber":902}

,

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"orchestrateStart","fileName":"VirtualMachineManagerImpl.java","lineNumber":4736}

,

{"declaringClass":"sun.reflect.NativeMethodAccessorImpl","methodName":"invoke0","fileName":"NativeMethodAccessorImpl.java","lineNumber":-2}

,

{"declaringClass":"sun.reflect.NativeMethodAccessorImpl","methodName":"invoke","fileName":"NativeMethodAccessorImpl.java","lineNumber":57}

,

{"declaringClass":"sun.reflect.DelegatingMethodAccessorImpl","methodName":"invoke","fileName":"DelegatingMethodAccessorImpl.java","lineNumber":43}

,

{"declaringClass":"java.lang.reflect.Method","methodName":"invoke","fileName":"Method.java","lineNumber":606}

,

{"declaringClass":"com.cloud.vm.VmWorkJobHandlerProxy","methodName":"handleVmWorkJob","fileName":"VmWorkJobHandlerProxy.java","lineNumber":107}

,

{"declaringClass":"com.cloud.vm.VirtualMachineManagerImpl","methodName":"handleVmWorkJob","fileName":"VirtualMachineManagerImpl.java","lineNumber":4855}

,

{"declaringClass":"com.cloud.vm.VmWorkJobDispatcher","methodName":"runJob","fileName":"VmWorkJobDispatcher.java","lineNumber":99}

,

{"declaringClass":"org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5","methodName":"runInContext","fileName":"AsyncJobManagerImpl.java","lineNumber":524}

,

{"declaringClass":"org.apache.cloudstack.managed.context.ManagedContextRunnable$1","methodName":"run","fileName":"ManagedContextRunnable.java","lineNumber":49}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1","methodName":"call","fileName":"DefaultManagedContext.java","lineNumber":56}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext","methodName":"callWithContext","fileName":"DefaultManagedContext.java","lineNumber":103}

,

{"declaringClass":"org.apache.cloudstack.managed.context.impl.DefaultManagedContext","methodName":"runWithContext","fileName":"DefaultManagedContext.java","lineNumber":53}

,

{"declaringClass":"org.apache.cloudstack.managed.context.ManagedContextRunnable","methodName":"run","fileName":"ManagedContextRunnable.java","lineNumber":46}

,

{"declaringClass":"java.util.concurrent.Executors$RunnableAdapter","methodName":"call","fileName":"Executors.java","lineNumber":471}

,

{"declaringClass":"java.util.concurrent.FutureTask$Sync","methodName":"innerRun","fileName":"FutureTask.java","lineNumber":334}

,

{"declaringClass":"java.util.concurrent.FutureTask","methodName":"run","fileName":"FutureTask.java","lineNumber":166}

,

{"declaringClass":"java.util.concurrent.ThreadPoolExecutor","methodName":"runWorker","fileName":"ThreadPoolExecutor.java","lineNumber":1145}

,

{"declaringClass":"java.util.concurrent.ThreadPoolExecutor$Worker","methodName":"run","fileName":"ThreadPoolExecutor.java","lineNumber":615}

,

{"declaringClass":"java.lang.Thread","methodName":"run","fileName":"Thread.java","lineNumber":724}

],"suppressedExceptions":[]}
at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.fromObjectSerializedString(JobSerializerHelper.java:135)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl.unmarshallResultObject(AsyncJobManagerImpl.java:669)
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:758)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:592)
at org.apache.cloudstack.engine.cloud.entity.api.VMEntityManagerImpl.deployVirtualMachine(VMEntityManagerImpl.java:237)
at org.apache.cloudstack.engine.cloud.entity.api.VirtualMachineEntityImpl.deploy(VirtualMachineEntityImpl.java:207)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3572)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3152)
at com.cloud.vm.UserVmManagerImpl.startVirtualMachine(UserVmManagerImpl.java:3138)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:50)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at com.sun.proxy.$Proxy212.startVirtualMachine(Unknown Source)
at org.apache.cloudstack.api.command.user.vm.DeployVMCmd.execute(DeployVMCmd.java:443)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:524)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.StreamCorruptedException: invalid stream header: 72899C96
at java.io.ObjectInputStream.readStreamHeader(ObjectInputStream.java:802)
at java.io.ObjectInputStream.<init>(ObjectInputStream.java:299)
at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.fromObjectSerializedString(JobSerializerHelper.java:129)
... 41 more
INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-4:ctx-43a4d668) Remove job-45 from job monitoring
WARN [o.a.c.alerts] (CapacityChecker:ctx-6ebd16d4) alertType:: 24 // dataCenterId:: 1 // podId:: null // clusterId:: null // message:: System Alert: Number of unallocated shared network IPs is low in availability zone NEWZONE