[CLOUDSTACK-6004] Cannot do Local to Local Xen StorageMotion using XS6.2SP1

• Type: Bug
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Affects Version/s: 4.2.1
• Fix Version/s: 4.5.0
• Component/s: Hypervisor Controller, Management Server
• Security Level: Public (Anyone can view this level - this is the default.)
• Assignee: Unassigned
• Reporter: Francois Gaudreault
• Created: 31/Jan/14 15:21
• Updated: 14/Jan/16 07:29
• Resolved: 13/Mar/15 03:05

Description

Filed as a blocker since it impacts the ability to properly do operational maintenances without downtime.

When we want to migrate a VM from one host with local storage to another, the migration fails with the following Xen exception:

VM_REQUIRES_NETWORK

That means the network is not online on the destination node. If the destination already have running VMs on the isolated network, the exception is not sent, but the VM will stay in the migrating state forever, nothing happens basically.

Stack trace:
2014-01-28 20:29:39,891 DEBUG [agent.transport.Request] (Job-Executor-13:job-61 = [ 01d36458-1b0a-4b62-998f-76e9a245ae78 ]) Seq 3-569311789: Sending { Cmd , MgmtId: 99553008314826, via: 3, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.MigrateWithStorageCommand":{"vm":{"id":24,"name":"i-5-24-VM","bootloader":"PyGrub","type":"User","cpus":1,"minSpeed":2000,"maxSpeed":2000,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS 5.3 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"d4a09176976a0d1c","params":

{"platform":"viridian:true;acpi:1;apic:true;pae:true;nx:true"}

,"uuid":"4b2074e9-2546-4124-ab89-0436fde8c74b","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"44990af7-3b20-42b6-9b34-5d26d8c79a07","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"7b1f10b6-c28a-e22a-8e1a-ce7472da3abd","id":2,"poolType":"EXT","host":"10.110.0.32","path":"ext","port":0}},"name":"ROOT-24","size":21474836480,"path":"45fe1cc1-101e-404b-815b-349b81bfbf85","volumeId":30,"vmName":"i-5-24-VM","accountId":5,"format":"VHD","id":30,"hypervisorType":"XenServer"}},"diskSeq":0,"type":"ROOT"}],"nics":[

{"deviceId":0,"networkRateMbps":-1,"defaultNic":true,"uuid":"0c935675-1446-4c11-b53b-a0aa4f6df5f0","ip":"10.1.1.246","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:16:ca:00:05","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1197","isolationUri":"vlan://1197","isSecurityGroupEnabled":false,"name":"Data"}

]},"volumeToFiler":{"Vol[30|ROOT|45fe1cc1-101e-404b-815b-349b81bfbf85|21474836480]":{"id":1,"uuid":"1c5adfea-61fc-9350-6e1b-62b55b5e7ed8","host":"10.110.0.30","path":"ext","port":0,"type":"EXT"}},"wait":0}}] }
2014-01-28 20:29:39,891 DEBUG [agent.transport.Request] (Job-Executor-13:job-61 = [ 01d36458-1b0a-4b62-998f-76e9a245ae78 ]) Seq 3-569311789: Executing: { Cmd , MgmtId: 99553008314826, via: 3, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.MigrateWithStorageCommand":{"vm":{"id":24,"name":"i-5-24-VM","bootloader":"PyGrub","type":"User","cpus":1,"minSpeed":2000,"maxSpeed":2000,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS 5.3 (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"d4a09176976a0d1c","params":

{"platform":"viridian:true;acpi:1;apic:true;pae:true;nx:true"}

,"uuid":"4b2074e9-2546-4124-ab89-0436fde8c74b","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"44990af7-3b20-42b6-9b34-5d26d8c79a07","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"7b1f10b6-c28a-e22a-8e1a-ce7472da3abd","id":2,"poolType":"EXT","host":"10.110.0.32","path":"ext","port":0}},"name":"ROOT-24","size":21474836480,"path":"45fe1cc1-101e-404b-815b-349b81bfbf85","volumeId":30,"vmName":"i-5-24-VM","accountId":5,"format":"VHD","id":30,"hypervisorType":"XenServer"}},"diskSeq":0,"type":"ROOT"}],"nics":[

{"deviceId":0,"networkRateMbps":-1,"defaultNic":true,"uuid":"0c935675-1446-4c11-b53b-a0aa4f6df5f0","ip":"10.1.1.246","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:16:ca:00:05","dns1":"8.8.8.8","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1197","isolationUri":"vlan://1197","isSecurityGroupEnabled":false,"name":"Data"}

]},"volumeToFiler":{"Vol[30|ROOT|45fe1cc1-101e-404b-815b-349b81bfbf85|21474836480]":{"id":1,"uuid":"1c5adfea-61fc-9350-6e1b-62b55b5e7ed8","host":"10.110.0.30","path":"ext","port":0,"type":"EXT"}},"wait":0}}] }
2014-01-28 20:29:39,892 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-166:null) Seq 3-569311789: Executing request
2014-01-28 20:29:39,944 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 7-685375517: Received: { Ans: , MgmtId: 99553008314826, via: 7, Ver: v1, Flags: 10,

{ GetStorageStatsAnswer } }
2014-01-28 20:29:39,952 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-101:null) Seq 1-612237852: Executing request
2014-01-28 20:29:40,073 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-166:null) SR retrieved for 1c5adfea-61fc-9350-6e1b-62b55b5e7ed8
2014-01-28 20:29:40,079 DEBUG [xen.resource.CitrixResourceBase] (DirectAgent-166:null) Checking 1c5adfea-61fc-9350-6e1b-62b55b5e7ed8 or SR 1c5adfea-61fc-9350-6e1b-62b55b5e7ed8 on XS[bfb8f492-826a-4c13-8822-ce1a81f03369-10.110.0.31]
2014-01-28 20:29:40,311 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-101:null) Seq 1-612237852: Response Received:
2014-01-28 20:29:40,312 DEBUG [agent.transport.Request] (StatsCollector-1:null) Seq 1-612237852: Received: { Ans: , MgmtId: 99553008314826, via: 1, Ver: v1, Flags: 10, { GetStorageStatsAnswer }

}
2014-01-28 20:29:40,318 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-348:null) Seq 2-141755018: Executing request
2014-01-28 20:29:41,097 WARN [xen.resource.CitrixResourceBase] (DirectAgent-166:null) Task failed! Task record: uuid: 7006501a-aaa4-e3ee-0cf5-a8f5d4b70ec1
nameLabel: Async.VM.assert_can_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Tue Jan 28 20:29:40 EST 2014
finished: Tue Jan 28 20:29:40 EST 2014
status: failure
residentOn: com.xensource.xenapi.Host@f129c335
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK, OpaqueRef:749414bf-aa6e-74ef-6458-1023998f20e4, OpaqueRef:b396c850-ec81-165e-560e-07e4ff41559e]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []

2014-01-28 20:29:41,101 WARN [xen.resource.XenServer610Resource] (DirectAgent-166:null) Catch Exception com.xensource.xenapi.Types$BadAsyncResult. Storage motion failed due to Task failed! Task record: uuid: 7006501a-aaa4-e3ee-0cf5-a8f5d4b70ec1
nameLabel: Async.VM.assert_can_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Tue Jan 28 20:29:40 EST 2014
finished: Tue Jan 28 20:29:40 EST 2014
status: failure
residentOn: com.xensource.xenapi.Host@f129c335
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK, OpaqueRef:749414bf-aa6e-74ef-6458-1023998f20e4, OpaqueRef:b396c850-ec81-165e-560e-07e4ff41559e]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []

Task failed! Task record: uuid: 7006501a-aaa4-e3ee-0cf5-a8f5d4b70ec1
nameLabel: Async.VM.assert_can_migrate
nameDescription:
allowedOperations: []
currentOperations: {}
created: Tue Jan 28 20:29:40 EST 2014
finished: Tue Jan 28 20:29:40 EST 2014
status: failure
residentOn: com.xensource.xenapi.Host@f129c335
progress: 1.0
type: <none/>
result:
errorInfo: [VM_REQUIRES_NETWORK, OpaqueRef:749414bf-aa6e-74ef-6458-1023998f20e4, OpaqueRef:b396c850-ec81-165e-560e-07e4ff41559e]
otherConfig: {}
subtaskOf: com.xensource.xenapi.Task@aaf13f6f
subtasks: []

at com.cloud.hypervisor.xen.resource.CitrixResourceBase.checkForSuccess(CitrixResourceBase.java:3648)
at com.cloud.hypervisor.xen.resource.XenServer610Resource.execute(XenServer610Resource.java:182)
at com.cloud.hypervisor.xen.resource.XenServer610Resource.executeRequest(XenServer610Resource.java:96)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:701)