[CLOUDSTACK-8412] VM migration with storage fails in a clustered management server setup - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Critical
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 4.6.0
Component/s: None
Security Level: Public (Anyone can view this level - this is the default.)
Labels:
None

Description

When a VM migration with storage request is forwarded by CS from one MS to another, de-serialization of the JSON request will fail in second management server with the below error -

2015-04-18 10:43:33,232 INFO  [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-52:ctx-d5bbb06c job-120140) (logid:88d08ca3) Add job-120140 into job monitoring
2015-04-18 10:43:33,245 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-52:ctx-d5bbb06c job-120140) (logid:58b8c3e1) Executing AsyncJobVO {id:120140, userId: 42, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd, cmdInfo: {"virtualmachineid":"cf879643-3143-4822-a727-0223783d0553","cmdEventType":"VM.MIGRATE","ctxUserId":"42","httpmethod":"GET","migrateto[0].pool":"f04a44e1-d75e-3829-813d-8bb6f356f3c2","migrateto[0].volume":"933adc86-44aa-4afd-b704-d5572f513a13","apiKey":"LrQgAYP2C8u47ZnFahkISur_4eKlWLTZK7fnkOJw5AuMtgz3s0KFKP1d6ow_mx5ecbI3zS8w18fG-ljIZ2m5dA","response":"json","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":\"cf879643-3143-4822-a727-0223783d0553\",\"com.cloud.host.Host\":\"9a74034e-2dbd-4b4f-84e3-9a7d828fdcb4\"}","hostid":"9a74034e-2dbd-4b4f-84e3-9a7d828fdcb4","ctxAccountId":"2","ctxStartEventId":"64200","signature":"gmEZXBo5X0Lwcc4fQ9MAgvsAoW8\u003d"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345049612948, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-04-18 10:43:33,245 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-17:ctx-53d5ddc1 ctx-7d9c3e95 ctx-ba112db2) (logid:69812f7a) submit async job-120140, details: AsyncJobVO {id:120140, userId: 42, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd, cmdInfo: {"virtualmachineid":"cf879643-3143-4822-a727-0223783d0553","cmdEventType":"VM.MIGRATE","ctxUserId":"42","httpmethod":"GET","migrateto[0].pool":"f04a44e1-d75e-3829-813d-8bb6f356f3c2","migrateto[0].volume":"933adc86-44aa-4afd-b704-d5572f513a13","apiKey":"LrQgAYP2C8u47ZnFahkISur_4eKlWLTZK7fnkOJw5AuMtgz3s0KFKP1d6ow_mx5ecbI3zS8w18fG-ljIZ2m5dA","response":"json","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":\"cf879643-3143-4822-a727-0223783d0553\",\"com.cloud.host.Host\":\"9a74034e-2dbd-4b4f-84e3-9a7d828fdcb4\"}","hostid":"9a74034e-2dbd-4b4f-84e3-9a7d828fdcb4","ctxAccountId":"2","ctxStartEventId":"64200","signature":"gmEZXBo5X0Lwcc4fQ9MAgvsAoW8\u003d"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 345049612948, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2015-04-18 10:43:33,301 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-52:ctx-d5bbb06c job-120140 ctx-36e40c6e) (logid:58b8c3e1) Sync job-120141 execution on object VmWorkJobQueue.2082
2015-04-18 10:43:33,985 ERROR [c.c.a.t.Request] (AgentManager-Handler-1:null) (logid:) Caught problem with [{"com.cloud.agent.api.MigrateWithStorageCommand":{"vm":{"id":2082,"name":"i-3-2082-VM","bootloader":"HVM","type":"User","cpus":2,"minSpeed":1860,"maxSpeed":1860,"minRam":4294967296,"maxRam":4294967296,"hostName":"PKNSRV-DC01","arch":"x86_64","os":"Windows Server 2012 R2 (64-bit)","platformEmulator":"windows8Server64Guest","bootArgs":"","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":true,"vncPassword":"f1ce1313c3a2ce1c","params":{"memoryOvercommitRatio":"1","Message.ReservedCapacityFreed.Flag":"true","dataDiskController":"osdefault","cpuOvercommitRatio":"4.0","nicAdapter":"Vmxnet3","vmware.reserve.cpu":"false","nestedVirtualizationFlag":"false","rootDiskController":"osdefault","vmware.reserve.mem":"false"},"uuid":"cf879643-3143-4822-a727-0223783d0553","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"933adc86-44aa-4afd-b704-d5572f513a13","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f14aba0e-cd98-3397-81da-a8e00f23f1eb","id":216,"poolType":"VMFS","host":"VMFS datastore: /REALCLOUD/RCBLNSAN03-VMCLP-Vol02","path":"/REALCLOUD/RCBLNSAN03-VMCLP-Vol02","port":0,"url":"VMFS://VMFS datastore: /REALCLOUD/RCBLNSAN03-VMCLP-Vol02/REALCLOUD/RCBLNSAN03-VMCLP-Vol02/?ROLE\u003dPrimary\u0026STOREUUID\u003df14aba0e-cd98-3397-81da-a8e00f23f1eb"}},"name":"ROOT-2082","size":42949672960,"path":"ROOT-2082-000001","volumeId":2790,"vmName":"i-3-2082-VM","accountId":3,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[RCBLNSAN03-VMCLP-Vol02] i-3-2082-VM/ROOT-2082-000001.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":2790,"deviceId":0,"cacheMode":"NONE","hypervisorType":"VMware"}},"diskSeq":0,"path":"ROOT-2082-000001","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"VMFS datastore: /REALCLOUD/RCBLNSAN03-VMCLP-Vol02","volumeSize":"42949672960"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":1000,"defaultNic":true,"pxeDisable":false,"nicUuid":"e6cab6cc-4378-4797-9dbf-e901d7d6c806","uuid":"aa7ff2c7-738e-47dc-bdaf-8e081c63053d","ip":"10.10.1.8","netmask":"255.255.248.0","gateway":"10.10.4.4","mac":"06:92:2a:00:05:dc","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2","isolationUri":"vlan://2","isSecurityGroupEnabled":false,"name":"vSwitch2"}]},"volumeToFiler":{"Vol[2790|ROOT|ROOT-2082-000001|42949672960]":{"id":244,"uuid":"f04a44e1-d75e-3829-813d-8bb6f356f3c2","host":"VMFS datastore: /REALCLOUD/RCBLNEMC01-VMCLP-Vol05","path":"/REALCLOUD/RCBLNEMC01-VMCLP-Vol05","port":0,"type":"VMFS"}},"tgtHost":"HostSystem:host-25811@10.90.2.11","contextMap":{"job":"job-120140/job-120141","logid":"58b8c3e1"},"wait":0}}]
com.google.gson.JsonParseException: Expecting object found: "Vol[2790|ROOT|ROOT-2082-000001|42949672960]"
        at com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
        at com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
        at com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
        at com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
        at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
        at com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:521)
        at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
        at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
        at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
        at com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:521)
        at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
        at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
        at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:498)
        at com.cloud.agent.transport.Request.getCommands(Request.java:249)
        at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:347)
        at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:656)
        at com.cloud.utils.nio.Task.run(Task.java:84)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
2015-04-18 10:43:33,986 WARN  [c.c.u.n.Task] (AgentManager-Handler-1:null) (logid:) Caught the following exception but pushing on
com.google.gson.JsonParseException: Expecting object found: "Vol[2790|ROOT|ROOT-2082-000001|42949672960]"
        at com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
        at com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
        at com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
        at com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
        at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
        at com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
        at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:521)
        at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
        at com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
        at com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
        at com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
        at com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
        at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
        at com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
        at com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
at com.google.gson.Gson.fromJson(Gson.java:551)
        at com.google.gson.Gson.fromJson(Gson.java:498)
        at com.cloud.agent.transport.Request.getCommands(Request.java:249)
        at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:347)
        at com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:656)
        at com.cloud.utils.nio.Task.run(Task.java:84)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)

Attachments

Activity

People

Assignee:: Likitha Shetty

Reporter:: Likitha Shetty

Votes:: 0 Vote for this issue

Watchers:: 2 Start watching this issue

Dates

Created:: 28/Apr/15 04:49

Updated:: 29/Apr/15 14:51

Resolved:: 28/Apr/15 05:03