[CLOUDSTACK-7078] CLONE - [VMWARE]System VM's are failed to start with Nexus enabled Zone

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.4.0
• Fix Version/s: 4.5.0
• Component/s: Network Devices, VMware
• Security Level: Public (Anyone can view this level - this is the default.)
• Assignee: Sateesh Chodapuneedi
• Reporter: Sateesh Chodapuneedi
• Created: 08/Jul/14 12:22
• Updated: 31/Jul/14 18:17
• Resolved: 31/Jul/14 18:17

Description

Steps:

1. Upgraded from 3.0.7 to 4.2 ( VMWARE Zone with Standard vSwitch)
2. Enable Nexus global config variable
3. Tried to add new Zone with VMWARE Nexus enabled
4. Added two physical networks ( 1 - Mgmt - vSwitch0,,vmwaresvs 2- Public & guest -> sailajanewpp,,nexusdvs)
5. Provided VSM details while adding cluster

Observation:
System VM's are failed to start with Nexus enabled Zone saying Nexus details can not be retrieved from DB.

2013-08-22 00:37:01,732 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) ===START=== 10.101.255.30 – GET command=queryAsyncJobResult&jobId=921f4c19-a655-4234-82c4-66efcbe73933&response=json&sessionkey=CNY1grUrySJwTE%2BpFM3btn1i4Xs%3D&_=1377112271133
2013-08-22 00:37:01,782 DEBUG [cloud.api.ApiServlet] (catalina-exec-2:null) ===END=== 10.101.255.30 – GET command=queryAsyncJobResult&jobId=921f4c19-a655-4234-82c4-66efcbe73933&response=json&sessionkey=CNY1grUrySJwTE%2BpFM3btn1i4Xs%3D&_=1377112271133
2013-08-22 00:37:02,170 WARN [vmware.resource.VmwareResource] (DirectAgent-128:10.102.192.18) StartCommand failed due to Exception: java.lang.Exception
Message: Failed to retrieve required credentials of Nexus VSM from database.

java.lang.Exception: Failed to retrieve required credentials of Nexus VSM from database.
at com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.getValidatedVsmCredentials(HypervisorHostHelper.java:183)
at com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.createPortProfile(HypervisorHostHelper.java:201)
at com.cloud.hypervisor.vmware.mo.HypervisorHostHelper.prepareNetwork(HypervisorHostHelper.java:584)
at com.cloud.hypervisor.vmware.resource.VmwareResource.prepareNetworkFromNicInfo(VmwareResource.java:3308)
at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:2904)
at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:514)
at com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-08-22 00:37:02,196 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-128:null) Seq 14-773455885: Cancelling because one of the answers is false and it is stop on error.
2013-08-22 00:37:02,196 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-128:null) Seq 14-773455885: Response Received:
2013-08-22 00:37:02,199 DEBUG [agent.transport.Request] (DirectAgent-128:null) Seq 14-773455885: Processing: { Ans: , MgmtId: 187767034175903, via: 14, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StartAnswer":{"vm":{"id":30,"name":"s-30-VM","bootloader":"HVM","type":"SecondaryStorageVm","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"hostName":"s-30-VM","arch":"x86_64","os":"Debian GNU/Linux 5.0 (32-bit)","bootArgs":" template=domP type=secstorage host=10.102.192.207 port=8250 name=s-30-VM zone=3 pod=3 guid=s-30-VM resource=com.cloud.storage.resource.PremiumSecondaryStorageResource instance=SecStorage sslcopy=true role=templateProcessor mtu=1500 eth2ip=10.102.196.220 eth2mask=255.255.255.0 gateway=10.102.196.1 public.network.device=eth2 eth0mask=0.0.0.0 eth0ip=0.0.0.0 eth1ip=10.102.195.150 eth1mask=255.255.252.0 mgmtcidr=10.102.192.0/22 localgw=10.102.192.1 private.network.device=eth1 eth3ip=10.102.195.152 eth3mask=255.255.252.0 storageip=10.102.195.152 storagenetmask=255.255.252.0 storagegateway=10.102.192.1 internaldns1=10.103.128.15 dns1=10.103.128.15","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"dd6c6644e27dee4b","params":

{"nicAdapter":"E1000"}

,"uuid":"dfa9a886-a75a-40a1-8d1b-8405560f4a8e","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c2eb619c-7200-4ece-9d92-4f5604891d0d","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"6e1b7ef2-1715-3f55-abf3-8226a6f13db6","id":206,"poolType":"NetworkFilesystem","host":"10.102.192.100","path":"/cpg_vol/sailaja/vnmcnewps1","port":2049}},"name":"ROOT-30","size":0,"path":"ROOT-30","volumeId":83,"vmName":"s-30-VM","accountId":1,"format":"OVA","id":83,"hypervisorType":"VMware"}},"diskSeq":0,"type":"ROOT"}],"nics":[

{"deviceId":2,"networkRateMbps":-1,"defaultNic":true,"uuid":"62a3ff0d-00f9-4491-8222-a1a503a14cd7","ip":"10.102.196.220","netmask":"255.255.255.0","gateway":"10.102.196.1","mac":"06:2f:ea:00:00:07","dns1":"10.103.128.15","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://100","isolationUri":"vlan://100","isSecurityGroupEnabled":false,"name":"sailajanewpp,,nexusdvs"}

,

{"deviceId":0,"networkRateMbps":-1,"defaultNic":false,"uuid":"15f068e3-93da-48e0-95fd-85ab7534a5f2","mac":"02:00:2f:89:00:11","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false,"name":"vSwitch0"}

,

{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"uuid":"645b6314-a9b3-4cea-8e59-36c375220762","ip":"10.102.195.150","netmask":"255.255.252.0","gateway":"10.102.192.1","mac":"06:ed:d0:00:00:01","broadcastType":"Native","type":"Management","isSecurityGroupEnabled":false,"name":"vSwitch0"}

,

{"deviceId":3,"networkRateMbps":-1,"defaultNic":false,"uuid":"b15358b4-19fa-4380-acdd-bbad598ccbc4","ip":"10.102.195.152","netmask":"255.255.252.0","gateway":"10.102.192.1","mac":"06:fc:36:00:00:03","broadcastType":"Native","type":"Storage","isSecurityGroupEnabled":false,"name":"vSwitch0"}

]},"result":false,"details":"StartCommand failed due to Exception: java.lang.Exception\nMessage: Failed to retrieve required credentials of Nexus VSM from database.\n","wait":0}}] }
2013-08-22 00:37:02,203 DEBUG [agent.transport.Request] (Job-Executor-1:job-81 = [ 921f4c19-a655-4234-82c4-66efcbe73933 ]) Seq 14-773455885: Received: { Ans: , MgmtId: 187767034175903, via: 14, Ver: v1, Flags: 10,

{ StartAnswer }

}
2013-08-22 00:37:02,249 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-81 = [ 921f4c19-a655-4234-82c4-66efcbe73933 ]) Unable to start VM on Host[-14-Routing] due to StartCommand failed due to Exception: java.lang.Exception
Message: Failed to retrieve required credentials of Nexus VSM from database.

2013-08-22 00:37:02,294 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1:job-81 = [ 921f4c19-a655-4234-82c4-66efcbe73933 ]) Cleaning up resources for the vm VM[SecondaryStorageVm|s-30-VM] in Starting state
@

Issue Links

• is a clone of: CLOUDSTACK-4435 [VMWARE]System VM's are failed to start with Nexus enabled Zone