[CLOUDSTACK-3441] [Load Test] High delays between VM being allocated to Pod and network implementation causing delays in VM deployment

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.2.0
• Fix Version/s: 4.2.0
• Component/s: Management Server, Network Controller
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: Load test, with simulator setup
• Assignee: Koushik Das
• Reporter: Sowmya Krishnan
• Created: 10/Jul/13 09:47
• Updated: 22/Aug/13 22:40
• Resolved: 21/Aug/13 09:57

Description

Scale Test set up with 20K simulated hosts, 20K VMs, basic zone, 3 Management servers
Also using security groups, ~5K VMs are deployed in every Security group.

While deployment of simulator VMs, following is observed after around 4K VMs:

There's delay between VM being allocated to Pod and network implementation causing delays in VM deployment. Following are the logs:

2013-06-10 15:51:01,215 DEBUG [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-1004:job-3014) VM is being created in podId: 1390
2013-06-10 15:55:31,494 DEBUG [cloud.network.NetworkManagerImpl] (Job-Executor-1004:job-3014) Lock is acquired for network id 204 as a part of network implement

This causes delay in VM deployment and the async job to complete. The delays get higher and higher as more and more VMs are deployed causing an unacceptable delay for huge deployments, viz, > 15-20K VMs