[CLOUDSTACK-6868] [Hyper-V] Download of an uploaded volume is failing

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.4.0
• Fix Version/s: 4.4.0
• Component/s: Management Server, Storage Controller
• Security Level: Public (Anyone can view this level - this is the default.)
• Labels: hyper-V, hyper-v hyperv
• Environment: Hyper-V
• Assignee: Anshul Gangwar
• Reporter: Abhinav Roy
• Created: 09/Jun/14 08:33
• Updated: 16/Jun/14 10:25
• Resolved: 13/Jun/14 10:54

Description

Steps:
=============
1. Deploy an advanced zone Hyper-V setup.
2. Create a VM v1
3. Upload a volume upload1
4. Download upload1

Expected behavior :
=============
Download should pass.

Observed behavior :
=============
Download fails with

2014-06-09 14:06:16,740 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-2b2fe04f) ===START=== 10.144.7.13 – GET command=extractVolume&id=7282784d-75bf-46a4-b6a5-33fae5cda75c&zoneid=622e7c06-e62e-4a6b-8de6-a462a3ee7cf7&mode=HTTP_DOWNLOAD&response=json&sessionkey=RGKWL0km%2FibmrEZH39fWJsmJeeU%3D&_=1402302332046
2014-06-09 14:06:16,783 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-13:ctx-2b2fe04f ctx-676b8816) submit async job-113, details: AsyncJobVO {id:113, userId: 2, accountId: 2, instanceType: Volume, instanceId: 18, cmd: org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd, cmdInfo: {"response":"json","id":"7282784d-75bf-46a4-b6a5-33fae5cda75c","sessionkey":"RGKWL0km/ibmrEZH39fWJsmJeeU\u003d","ctxDetails":"

{\"com.cloud.storage.Volume\":18,\"Volume\":\"7282784d-75bf-46a4-b6a5-33fae5cda75c\",\"com.cloud.dc.DataCenter\":1}

","cmdEventType":"VOLUME.EXTRACT","ctxUserId":"2","zoneid":"622e7c06-e62e-4a6b-8de6-a462a3ee7cf7","httpmethod":"GET","_":"1402302332046","uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","ctxAccountId":"2","ctxStartEventId":"197","mode":"HTTP_DOWNLOAD"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 213737702773493, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-06-09 14:06:16,783 DEBUG [c.c.a.ApiServlet] (catalina-exec-13:ctx-2b2fe04f ctx-676b8816) ===END=== 10.144.7.13 – GET command=extractVolume&id=7282784d-75bf-46a4-b6a5-33fae5cda75c&zoneid=622e7c06-e62e-4a6b-8de6-a462a3ee7cf7&mode=HTTP_DOWNLOAD&response=json&sessionkey=RGKWL0km%2FibmrEZH39fWJsmJeeU%3D&_=1402302332046
2014-06-09 14:06:16,785 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-28:ctx-774021c1 job-113) Add job-113 into job monitoring
2014-06-09 14:06:16,786 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-28:ctx-774021c1 job-113) Executing AsyncJobVO {id:113, userId: 2, accountId: 2, instanceType: Volume, instanceId: 18, cmd: org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd, cmdInfo: {"response":"json","id":"7282784d-75bf-46a4-b6a5-33fae5cda75c","sessionkey":"RGKWL0km/ibmrEZH39fWJsmJeeU\u003d","ctxDetails":"

{\"com.cloud.storage.Volume\":18,\"Volume\":\"7282784d-75bf-46a4-b6a5-33fae5cda75c\",\"com.cloud.dc.DataCenter\":1}

","cmdEventType":"VOLUME.EXTRACT","ctxUserId":"2","zoneid":"622e7c06-e62e-4a6b-8de6-a462a3ee7cf7","httpmethod":"GET","_":"1402302332046","uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","ctxAccountId":"2","ctxStartEventId":"197","mode":"HTTP_DOWNLOAD"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 213737702773493, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2014-06-09 14:06:16,795 WARN [c.c.a.d.ParamGenericValidationWorker] (API-Job-Executor-28:ctx-774021c1 job-113 ctx-351bc29f) Received unknown parameters for command extractVolume. Unknown parameters : ctxdetails
2014-06-09 14:06:16,833 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (API-Job-Executor-28:ctx-774021c1 job-113 ctx-351bc29f) copyAsync inspecting src type VOLUME copyAsync inspecting dest type VOLUME
2014-06-09 14:06:16,850 DEBUG [c.c.a.t.Request] (API-Job-Executor-28:ctx-774021c1 job-113 ctx-351bc29f) Seq 1-9177491615650939381: Sending { Cmd , MgmtId: 213737702773493, via: 1(10.102.244.20), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"641da331-d6d6-3224-b53a-0dad93667d30","id":2,"poolType":"SMB","host":"SMB19","path":"/hyperv-share/abhinav-ps1?user=abhinavroy&domain=BLR","port":445,"url":"SMB://SMB19/hyperv-share/abhinav-ps1?user=abhinavroy&domain=BLR/?ROLE=Primary&STOREUUID=641da331-d6d6-3224-b53a-0dad93667d30"}},"name":"upload3","size":21474836480,"volumeId":18,"accountId":2,"format":"VHDX","id":18,"hypervisorType":"Hyperv"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"cifs://10.102.192.19/hyperv-share/abhinav-ss1?user=abhinavroy&domain=BLR","_role":"Image"}},"name":"upload3","size":21474836480,"path":"volumes/2/18","volumeId":18,"accountId":2,"format":"VHDX","id":18,"hypervisorType":"Hyperv"}},"executeInSequence":false,"options":{},"wait":10800}}] }
2014-06-09 14:06:16,852 DEBUG [c.c.a.t.Request] (API-Job-Executor-28:ctx-774021c1 job-113 ctx-351bc29f) Seq 1-9177491615650939381: Executing: { Cmd , MgmtId: 213737702773493, via: 1(10.102.244.20), Ver: v1, Flags: 100011, [{"org.apache.cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"641da331-d6d6-3224-b53a-0dad93667d30","id":2,"poolType":"SMB","host":"SMB19","path":"/hyperv-share/abhinav-ps1?user=abhinavroy&domain=BLR","port":445,"url":"SMB://SMB19/hyperv-share/abhinav-ps1?user=abhinavroy&domain=BLR/?ROLE=Primary&STOREUUID=641da331-d6d6-3224-b53a-0dad93667d30"}},"name":"upload3","size":21474836480,"volumeId":18,"accountId":2,"format":"VHDX","id":18,"hypervisorType":"Hyperv"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"cifs://10.102.192.19/hyperv-share/abhinav-ss1?user=abhinavroy&domain=BLR","_role":"Image"}},"name":"upload3","size":21474836480,"path":"volumes/2/18","volumeId":18,"accountId":2,"format":"VHDX","id":18,"hypervisorType":"Hyperv"}},"executeInSequence":false,"options":{},"wait":10800}}] }
2014-06-09 14:06:16,852 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-396:ctx-3fcd7b13) Seq 1-9177491615650939381: Executing request
2014-06-09 14:06:16,853 DEBUG [c.c.h.h.r.HypervDirectConnectResource] (DirectAgent-396:ctx-3fcd7b13) POST request to https://10.102.244.20:8250/api/HypervResource/org.apache.cloudstack.storage.command.CopyCommand with contents {"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"641da331-d6d6-3224-b53a-0dad93667d30","id":2,"poolType":"SMB","host":"SMB19","path":"/hyperv-share/abhinav-ps1?user=abhinavroy&domain=BLR","port":445,"url":"SMB://SMB19/hyperv-share/abhinav-ps1?user=abhinavroy&domain=BLR/?ROLE=Primary&STOREUUID=641da331-d6d6-3224-b53a-0dad93667d30"}},"name":"upload3","size":21474836480,"volumeId":18,"accountId":2,"format":"VHDX","id":18,"hypervisorType":"Hyperv"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"cifs://10.102.192.19/hyperv-share/abhinav-ss1?user=abhinavroy&domain=BLR","_role":"Image"}},"name":"upload3","size":21474836480,"path":"volumes/2/18","volumeId":18,"accountId":2,"format":"VHDX","id":18,"hypervisorType":"Hyperv"}},"executeInSequence":false,"options":{},"contextMap":{},"wait":10800}
2014-06-09 14:06:16,856 DEBUG [c.c.h.h.r.HypervDirectConnectResource] (DirectAgent-396:ctx-3fcd7b13) Sending cmd to https://10.102.244.20:8250/api/HypervResource/org.apache.cloudstack.storage.command.CopyCommand cmd data:{"srcTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","volumeType":"DATADISK","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"641da331-d6d6-3224-b53a-0dad93667d30","id":2,"poolType":"SMB","host":"SMB19","path":"/hyperv-share/abhinav-ps1?user=abhinavroy&domain=BLR","port":445,"url":"SMB://SMB19/hyperv-share/abhinav-ps1?user=abhinavroy&domain=BLR/?ROLE=Primary&STOREUUID=641da331-d6d6-3224-b53a-0dad93667d30"}},"name":"upload3","size":21474836480,"volumeId":18,"accountId":2,"format":"VHDX","id":18,"hypervisorType":"Hyperv"}},"destTO":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"7282784d-75bf-46a4-b6a5-33fae5cda75c","volumeType":"DATADISK","dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"cifs://10.102.192.19/hyperv-share/abhinav-ss1?user=abhinavroy&domain=BLR","_role":"Image"}},"name":"upload3","size":21474836480,"path":"volumes/2/18","volumeId":18,"accountId":2,"format":"VHDX","id":18,"hypervisorType":"Hyperv"}},"executeInSequence":false,"options":{},"contextMap":{},"wait":10800}
2014-06-09 14:06:16,937 DEBUG [c.c.h.h.r.HypervDirectConnectResource] (DirectAgent-396:ctx-3fcd7b13) POST response is [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Local template file missing from \\smb19\hyperv-share\abhinav-ps1\7282784d-75bf-46a4-b6a5-33fae5cda75c.vhdx","newData":null,"contextMap":{}}}]
2014-06-09 14:06:16,938 DEBUG [c.c.h.h.r.HypervDirectConnectResource] (DirectAgent-396:ctx-3fcd7b13) executeRequest received response [Lcom.cloud.agent.api.Answer;@477df73e
2014-06-09 14:06:16,938 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-396:ctx-3fcd7b13) Seq 1-9177491615650939381: Response Received:
2014-06-09 14:06:16,938 DEBUG [c.c.a.t.Request] (DirectAgent-396:ctx-3fcd7b13) Seq 1-9177491615650939381: Processing: { Ans: , MgmtId: 213737702773493, via: 1, Ver: v1, Flags: 10, [{"org.apache.cloudstack.storage.command.CopyCmdAnswer":{"result":false,"details":"Local template file missing from \\\\smb19\\hyperv-share\\abhinav-ps1\\7282784d-75bf-46a4-b6a5-33fae5cda75c.vhdx","wait":0}}] }
2014-06-09 14:06:16,938 DEBUG [c.c.a.t.Request] (API-Job-Executor-28:ctx-774021c1 job-113 ctx-351bc29f) Seq 1-9177491615650939381: Received: { Ans: , MgmtId: 213737702773493, via: 1, Ver: v1, Flags: 10,

{ CopyCmdAnswer }

}
2014-06-09 14:06:16,943 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (API-Job-Executor-28:ctx-774021c1 job-113 ctx-351bc29f) Unsupported data object (VOLUME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@25eff6cc), no need to delete from object in store ref table
2014-06-09 14:06:16,964 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-28:ctx-774021c1 job-113) Unexpected exception while executing org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to copy the volume from the source primary storage pool to secondary storage.
at com.cloud.storage.VolumeApiServiceImpl.extractVolume(VolumeApiServiceImpl.java:1900)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy181.extractVolume(Unknown Source)
at org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd.execute(ExtractVolumeCmd.java:137)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:141)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:503)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:460)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2014-06-09 14:06:16,966 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-28:ctx-774021c1 job-113) Complete async job-113, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/

{"uuidList":[],"errorcode":530,"errortext":"Failed to copy the volume from the source primary storage pool to secondary storage."}

2014-06-09 14:06:16,982 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-28:ctx-774021c1 job-113) Done executing org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd for job-113

NOTE :
==============
The volume was uploaded as .vhd. In the primary storage also we see that the type is .vhd but in DB it is stored as .vhdx

mysql> select * from volumes where id=18 \G;

• • • • • • • • • • • • 1. row *********
                        id: 18
                        account_id: 2
                        domain_id: 1
                        pool_id: 2
                        last_pool_id: NULL
                        instance_id: NULL
                        device_id: NULL
                        name: upload3
                        uuid: 7282784d-75bf-46a4-b6a5-33fae5cda75c
                        size: 21474836480
                        folder: NULL
                        path: NULL
                        pod_id: NULL
                        data_center_id: 1
                        iscsi_name: NULL
                        host_ip: NULL
                        volume_type: DATADISK
                        pool_type: NULL
                        disk_offering_id: 6
                        template_id: NULL
                        first_snapshot_backup_uuid: NULL
                        recreatable: 0
                        created: 2014-06-09 06:42:37
                        attached: NULL
                        updated: 2014-06-09 06:51:29
                        removed: NULL
                        state: Ready
                        chain_info: NULL
                        update_count: 4
                        disk_type: NULL
                        vm_snapshot_chain_size: NULL
                        iso_id: NULL
                        display_volume: 1
                        format: VHDX
                        min_iops: NULL
                        max_iops: NULL
                        hv_ss_reserve: NULL
                        1 row in set (0.00 sec)