Details
Critical Description
steps :
====================
1. Deploy CS 2.2.13 advanced zone setup with KVM.
2. Create VMs, template,snapshots, domains, accounts etc
3. Upgrade the management server and agent to 2.2.14
4. Create VMs , templates, snapshots, domain, accounts etc.
5. Have some VMs in stopped state
6. Upgrade the management server and agent to 4.2
Expected behaviour :
===================
After upgrade we should be able to create volumes from the sanpshots which were created before upgrade
Observed behaviour:
===================
1. After upgrade i was able to create volumes from snapshots which were create after upgrade
2. But creation of volumes from the sanpshots which were created before upgrade is failing.
2013-08-07 16:38:39,245 DEBUG [storage.motion.AncientDataMotionStrategy] (Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) copyAsync inspecting src type SNAPSHOT copyAsync inspecting dest type VOLUME
2013-08-07 16:38:39,264 DEBUG [agent.transport.Request] (Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Seq 3-1516699752: Sending { Cmd , MgmtId: 226870599129537, via: 3, Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"snapshots/2/9//snapshots/1/2/9/i-2-9-VM_ROOT-9_20130805162236","volume":{"uuid":"9","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9fb7c84b-bfff-3ca2-9fc0-f50181c6b1f0","id":201,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhinav/kvm-pri2","port":2049}},"name":"ROOT-9","size":8589934592,"path":"c11c7a12-e9c7-495d-955b-6e814bb5b24f","volumeId":9,"vmName":"i-2-9-VM","accountId":2,"format":"QCOW2","id":9,"hypervisorType":"KVM"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.102.192.100/cpg_vol/abhinav/kvm-sec-old","_role":"Image"}},"vmName":"i-2-9-VM","name":"i-2-9-VM_ROOT-9_20130805162236","hypervisorType":"KVM","id":2}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"e3c99b55-b322-4c67-bef0-0dff0bd50e79","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"9fb7c84b-bfff-3ca2-9fc0-f50181c6b1f0","id":201,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/abhinav/kvm-pri2","port":2049}},"name":"volfromsnap-root7-2213","size":8589934592,"volumeId":34,"accountId":2,"format":"QCOW2","id":34,"hypervisorType":"None"}},"executeInSequence":false,"wait":10800}}] }
2013-08-07 16:38:39,467 DEBUG [agent.transport.Request] (AgentManager-Handler-6:null) Seq 3-1516699752: Processing: { Ans: , MgmtId: 226870599129537, via: 3, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error '/bin/mount 10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9 /mnt/790b44cf-2022-3b27-8270-390d8e309290' exited with non-zero status 32 and signal 0: mount.nfs: access denied by server while mounting 10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9\n","wait":0}}] }
2013-08-07 16:38:39,467 DEBUG [agent.transport.Request] (Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Seq 3-1516699752: Received: { Ans: , MgmtId: 226870599129537, via: 3, Ver: v1, Flags: 10,
}
2013-08-07 16:38:39,474 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@4dcf4da3), no need to delete from object in store ref table
2013-08-07 16:38:39,490 WARN [storage.datastore.ObjectInDataStoreManagerImpl] (Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Snapshot 2 is not found on image store 4, so no need to delete
2013-08-07 16:38:39,490 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Failed to create volume from snapshot:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error '/bin/mount 10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9 /mnt/790b44cf-2022-3b27-8270-390d8e309290' exited with non-zero status 32 and signal 0: mount.nfs: access denied by server while mounting 10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9
2013-08-07 16:38:39,491 DEBUG [cloud.storage.VolumeManagerImpl] (Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Failed to create volume: 34
com.cloud.utils.exception.CloudRuntimeException: Failed to create volume from snapshot:com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: internal error '/bin/mount 10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9 /mnt/790b44cf-2022-3b27-8270-390d8e309290' exited with non-zero status 32 and signal 0: mount.nfs: access denied by server while mounting 10.102.192.100:/cpg_vol/abhinav/kvm-sec-old/snapshots/2/9/snapshots/1/2/9
at com.cloud.storage.VolumeManagerImpl.createVolumeFromSnapshot(VolumeManagerImpl.java:558)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at com.cloud.storage.VolumeManagerImpl.createVolumeFromSnapshot(VolumeManagerImpl.java:603)
at com.cloud.storage.VolumeManagerImpl.createVolume(VolumeManagerImpl.java:1047)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at com.cloud.storage.VolumeManagerImpl.createVolume(VolumeManagerImpl.java:183)
at org.apache.cloudstack.api.command.user.volume.CreateVolumeCmd.execute(CreateVolumeCmd.java:189)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:636)
2013-08-07 16:38:39,521 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-10:job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ]) Complete async job-65 = [ db7e5dca-3771-4a82-a72d-574c7e96d5e6 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create a volume
Attaching all DB dumps, agent logs and management server logs.