[CLOUDSTACK-3774] [XenServer]NPE while creating template from snapshot on a upgraded setup

• Type: Bug
• Status: Closed
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 4.2.0
• Fix Version/s: 4.2.0
• Component/s: Storage Controller, Upgrade, XenServer
• Security Level: Public (Anyone can view this level - this is the default.)
• Assignee: Devdeep Singh
• Reporter: Sailaja Mada
• Created: 24/Jul/13 08:35
• Updated: 19/Aug/13 16:42
• Resolved: 06/Aug/13 11:27

Description

Setup: Xen 6.1 hosts , Two zone ( Zone1 with local storage ,Zone 2 - NFS )

Upgraded from 3.0.6 to 4.2

Steps:

1) Create Account X and deploy VM on a NFS storage Zone 3
2) Create snapshot from ROOT volume of this instance
3) Tried to create template from this snapshot.

Observations :

1. It failed with NPE.
2. UI results NPE as a failure message

2013-07-24 19:07:50,164 DEBUG [storage.resource.NfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) snapshots already exists on secondary storage, and is mounted at /var/lib/cloud/mnt/secStorage/0a639c32-c2df-36e3-883d-eebcb5093bf1
2013-07-24 19:07:50,164 INFO [storage.resource.NfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) snapshots directory created/exists on Secondary Storage.
2013-07-24 19:07:50,165 DEBUG [storage.resource.NfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) volumes already exists on secondary storage, and is mounted at /var/lib/cloud/mnt/secStorage/0a639c32-c2df-36e3-883d-eebcb5093bf1
2013-07-24 19:07:50,166 INFO [storage.resource.NfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) volumes directory created/exists on Secondary Storage.
2013-07-24 19:07:50,166 DEBUG [storage.resource.LocalNfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) Executing: sudo mount
2013-07-24 19:07:50,224 DEBUG [storage.resource.LocalNfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) Execution is successful.
2013-07-24 19:07:50,224 DEBUG [storage.resource.LocalNfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) mount point /var/lib/cloud/mnt/secStorage/0a639c32-c2df-36e3-883d-eebcb5093bf1 already exists
2013-07-24 19:07:50,309 DEBUG [storage.resource.NfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) Executing: null -p /var/lib/cloud/mnt/secStorage/0a639c32-c2df-36e3-883d-eebcb5093bf1/snapshots/4/40 -s b1988755-6c46-4c73-9d3c-2fd751d27825.vhd -n f29545c7-cb53-49c8-81a4-5e9ce566093a.vhd -t /var/lib/cloud/mnt/secStorage/0a639c32-c2df-36e3-883d-eebcb5093bf1/template/tmpl/4/207
2013-07-24 19:07:50,309 WARN [storage.resource.NfsSecondaryStorageResource] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) Exception: null -p /var/lib/cloud/mnt/secStorage/0a639c32-c2df-36e3-883d-eebcb5093bf1/snapshots/4/40 -s b1988755-6c46-4c73-9d3c-2fd751d27825.vhd -n f29545c7-cb53-49c8-81a4-5e9ce566093a.vhd -t /var/lib/cloud/mnt/secStorage/0a639c32-c2df-36e3-883d-eebcb5093bf1/template/tmpl/4/207
java.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:457)
at com.cloud.utils.script.Script.execute(Script.java:183)
at com.cloud.utils.script.Script.execute(Script.java:161)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.copySnapshotToTemplateFromNfsToNfsXenserver(NfsSecondaryStorageResource.java:360)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.copySnapshotToTemplateFromNfsToNfs(NfsSecondaryStorageResource.java:401)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.createTemplateFromSnapshot(NfsSecondaryStorageResource.java:419)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:445)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:208)
at org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.executeRequest(LocalNfsSecondaryStorageResource.java:76)
at org.apache.cloudstack.storage.LocalHostEndpoint.sendMessage(LocalHostEndpoint.java:93)
at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.createTemplateFromSnapshot(AncientDataMotionStrategy.java:384)
at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:344)
at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at org.apache.cloudstack.storage.image.TemplateServiceImpl.copyAsync(TemplateServiceImpl.java:549)
at org.apache.cloudstack.storage.image.TemplateServiceImpl.createTemplateFromSnapshotAsync(TemplateServiceImpl.java:556)
at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1355)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:263)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 19:07:50,348 DEBUG [cloud.template.TemplateManagerImpl] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) Failed to create templatejava.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:457)
at com.cloud.utils.script.Script.execute(Script.java:183)
at com.cloud.utils.script.Script.execute(Script.java:161)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.copySnapshotToTemplateFromNfsToNfsXenserver(NfsSecondaryStorageResource.java:360)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.copySnapshotToTemplateFromNfsToNfs(NfsSecondaryStorageResource.java:401)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.createTemplateFromSnapshot(NfsSecondaryStorageResource.java:419)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:445)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:208)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:208)
at org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.executeRequest(LocalNfsSecondaryStorageResource.java:76)
at org.apache.cloudstack.storage.LocalHostEndpoint.sendMessage(LocalHostEndpoint.java:93)
at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.createTemplateFromSnapshot(AncientDataMotionStrategy.java:384)
at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:344)
at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at org.apache.cloudstack.storage.image.TemplateServiceImpl.copyAsync(TemplateServiceImpl.java:549)
at org.apache.cloudstack.storage.image.TemplateServiceImpl.createTemplateFromSnapshotAsync(TemplateServiceImpl.java:556)
at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1355)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f.CGLIB$createPrivateTemplate$32(<generated>)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f_FastClassByCloudStack_738e8b.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f.createPrivateTemplate(<generated>)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:263)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

2013-07-24 19:07:50,381 ERROR [cloud.async.AsyncJobManagerImpl] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) Unexpected exception while executing org.apache.cloudstack.api.command.user.template.CreateTemplateCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to create templatejava.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:457)
at com.cloud.utils.script.Script.execute(Script.java:183)
at com.cloud.utils.script.Script.execute(Script.java:161)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.copySnapshotToTemplateFromNfsToNfsXenserver(NfsSecondaryStorageResource.java:360)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.copySnapshotToTemplateFromNfsToNfs(NfsSecondaryStorageResource.java:401)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.createTemplateFromSnapshot(NfsSecondaryStorageResource.java:419)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:445)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:208)
at org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.executeRequest(LocalNfsSecondaryStorageResource.java:76)
at org.apache.cloudstack.storage.LocalHostEndpoint.sendMessage(LocalHostEndpoint.java:93)
at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.createTemplateFromSnapshot(AncientDataMotionStrategy.java:384)
at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:344)
at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at org.apache.cloudstack.storage.image.TemplateServiceImpl.copyAsync(TemplateServiceImpl.java:549)
at org.apache.cloudstack.storage.image.TemplateServiceImpl.createTemplateFromSnapshotAsync(TemplateServiceImpl.java:556)
at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1355)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f.CGLIB$createPrivateTemplate$32(<generated>)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f_FastClassByCloudStack_738e8b.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f.createPrivateTemplate(<generated>)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:263)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1369)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:263)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-07-24 19:07:50,390 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-25:job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ]) Complete async job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create templatejava.lang.NullPointerException
at java.lang.ProcessBuilder.start(ProcessBuilder.java:457)
at com.cloud.utils.script.Script.execute(Script.java:183)
at com.cloud.utils.script.Script.execute(Script.java:161)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.copySnapshotToTemplateFromNfsToNfsXenserver(NfsSecondaryStorageResource.java:360)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.copySnapshotToTemplateFromNfsToNfs(NfsSecondaryStorageResource.java:401)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.createTemplateFromSnapshot(NfsSecondaryStorageResource.java:419)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.execute(NfsSecondaryStorageResource.java:445)
at org.apache.cloudstack.storage.resource.NfsSecondaryStorageResource.executeRequest(NfsSecondaryStorageResource.java:208)
at org.apache.cloudstack.storage.resource.LocalNfsSecondaryStorageResource.executeRequest(LocalNfsSecondaryStorageResource.java:76)
at org.apache.cloudstack.storage.LocalHostEndpoint.sendMessage(LocalHostEndpoint.java:93)
at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.createTemplateFromSnapshot(AncientDataMotionStrategy.java:384)
at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:344)
at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:55)
at org.apache.cloudstack.storage.image.TemplateServiceImpl.copyAsync(TemplateServiceImpl.java:549)
at org.apache.cloudstack.storage.image.TemplateServiceImpl.createTemplateFromSnapshotAsync(TemplateServiceImpl.java:556)
at com.cloud.template.TemplateManagerImpl.createPrivateTemplate(TemplateManagerImpl.java:1355)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f.CGLIB$createPrivateTemplate$32(<generated>)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f_FastClassByCloudStack_738e8b.invoke(<generated>)
at net.sf.cglib.proxy.MethodProxy.invokeSuper(MethodProxy.java:228)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at com.cloud.template.TemplateManagerImpl_EnhancerByCloudStack_9244e17f.createPrivateTemplate(<generated>)
at org.apache.cloudstack.api.command.user.template.CreateTemplateCmd.execute(CreateTemplateCmd.java:263)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)

2013-07-24 19:07:52,603 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===START=== 10.144.6.19 – GET command=queryAsyncJobResult&jobId=9d0965c2-23ce-4c82-abec-f68c5e523c56&response=json&sessionkey=AFany%2FrOuQtuA2%2FzjVw%2F4ryjKsc%3D&_=1374653618676
2013-07-24 19:07:52,629 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-15:null) Async job-85 = [ 9d0965c2-23ce-4c82-abec-f68c5e523c56 ] completed
2013-07-24 19:07:52,640 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) ===END=== 10.144.6.19 – GET command=queryAsyncJobResult&jobId=9d0965c2-23ce-4c82-abec-f68c5e523c56&response=json&sessionkey=AFany%2FrOuQtuA2%2FzjVw%2F4ryjKsc%3D&_=1374653618676
2013-07-24 19:07:52,769 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-11:null) SeqA 10-8916: Processing Seq 10-8916: { Cmd , MgmtId: -1, via: 10, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":18,"_loadInfo":"

{\n \"connections\": []\n}

","wait":0}}] }