[CLOUDSTACK-6423] No clusters found having a host with enough capacity

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.2.0, 4.2.1
• Fix Version/s: 4.2.0
• Component/s: Management Server
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: Centos 6.5 Master
• Assignee: Unassigned
• Reporter: Ameen Ali
• Created: 16/Apr/14 00:37
• Updated: 16/Apr/14 00:37

Description

I have just installed CloudStack 4.2.1 as management server on my Centos 6.5. I am having issues with starting the secondary storage VMs. Here are the messages from the log /var/log/cloudstack/management/management-server.log :

2014-04-16 03:22:00,789 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 routers to update status.
2014-04-16 03:22:00,790 DEBUG [network.router.VirtualNetworkApplianceManagerImpl] (RouterStatusMonitor-1:null) Found 0 networks to update RvR status.
2014-04-16 03:22:06,032 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Resetting hosts suitable for reconnect
2014-04-16 03:22:06,035 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed resetting hosts suitable for reconnect
2014-04-16 03:22:06,035 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Acquiring hosts for clusters already owned by this management server
2014-04-16 03:22:06,038 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed acquiring hosts for clusters already owned by this management server
2014-04-16 03:22:06,038 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Acquiring hosts for clusters not owned by any management server
2014-04-16 03:22:06,039 DEBUG [host.dao.HostDaoImpl] (ClusteredAgentManager Timer:null) Completed acquiring hosts for clusters not owned by any management server
2014-04-16 03:22:08,039 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-27:null) Ping from 1
2014-04-16 03:22:09,155 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-16:null) Seq 1-2066939909: Executing request
2014-04-16 03:22:09,434 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-16:null) Seq 1-2066939909: Response Received:
2014-04-16 03:22:09,435 DEBUG [agent.transport.Request] (DirectAgent-16:null) Seq 1-2066939909: Processing: { Ans: , MgmtId: 52240630156, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.ClusterSyncAnswer":{"_clusterId":1,"_newStates":{},"_isExecuted":false,"result":true,"wait":0}}] }
2014-04-16 03:22:28,685 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Zone 1 is ready to launch secondary storage VM
2014-04-16 03:22:28,687 INFO [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null) No running secondary storage vms found in datacenter id=1, starting one
2014-04-16 03:22:28,690 INFO [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) No stopped secondary storage vm is available, need to allocate a new secondary storage vm
2014-04-16 03:22:28,691 DEBUG [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Assign secondary storage vm from a newly started instance for request from data center : 1
2014-04-16 03:22:28,701 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Found existing network configuration for offering [Network Offering [6-Guest-DefaultSharedNetworkOfferingWithSGService]: Ntwk[204|Guest|6]
2014-04-16 03:22:28,702 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Releasing lock for Acct[63901c80-b520-11e3-83fd-000c29c8c18c-system]
2014-04-16 03:22:28,704 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Found existing network configuration for offering [Network Offering [3-Control-System-Control-Network]: Ntwk[202|Control|3]
2014-04-16 03:22:28,704 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Releasing lock for Acct[63901c80-b520-11e3-83fd-000c29c8c18c-system]
2014-04-16 03:22:28,706 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Found existing network configuration for offering [Network Offering [2-Management-System-Management-Network]: Ntwk[201|Management|2]
2014-04-16 03:22:28,706 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Releasing lock for Acct[63901c80-b520-11e3-83fd-000c29c8c18c-system]
2014-04-16 03:22:28,708 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Found existing network configuration for offering [Network Offering [4-Storage-System-Storage-Network]: Ntwk[203|Storage|4]
2014-04-16 03:22:28,708 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Releasing lock for Acct[63901c80-b520-11e3-83fd-000c29c8c18c-system]
2014-04-16 03:22:28,716 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Allocating entries for VM: VM[SecondaryStorageVm|s-316-VM]
2014-04-16 03:22:28,719 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Allocating nics for VM[SecondaryStorageVm|s-316-VM]
2014-04-16 03:22:28,719 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-316-VM] in network Ntwk[204|Guest|6] with requested profile NicProfile[0-0-null-null-null
2014-04-16 03:22:28,725 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-316-VM] in network Ntwk[202|Control|3] with requested profile null
2014-04-16 03:22:28,729 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-316-VM] in network Ntwk[201|Management|2] with requested profile null
2014-04-16 03:22:28,734 DEBUG [cloud.network.NetworkManagerImpl] (secstorage-1:null) Allocating nic for vm VM[SecondaryStorageVm|s-316-VM] in network Ntwk[203|Storage|4] with requested profile null
2014-04-16 03:22:28,739 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Allocating disks for VM[SecondaryStorageVm|s-316-VM]
2014-04-16 03:22:28,765 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Allocation completed for VM: VM[SecondaryStorageVm|s-316-VM]
2014-04-16 03:22:28,766 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter] (secstorage-1:null) received secondary storage vm alert
2014-04-16 03:22:28,767 DEBUG [cloud.alert.SecondaryStorageVmAlertAdapter] (secstorage-1:null) New secondary storage vm created, zone: Zone1, secStorageVm: s-316-VM, public IP: null, private IP: null
2014-04-16 03:22:28,771 DEBUG [cloud.capacity.CapacityManagerImpl] (secstorage-1:null) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2014-04-16 03:22:28,771 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Successfully transitioned to start state for VM[SecondaryStorageVm|s-316-VM] reservation id = ddfc829e-a6fc-4e39-9a5b-13244f1dd013
2014-04-16 03:22:28,774 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
2014-04-16 03:22:28,774 DEBUG [cloud.vm.VirtualMachineManagerImpl] (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2014-04-16 03:22:28,775 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (secstorage-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (secstorage-1:null) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_9c406fc8@6337c832
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (secstorage-1:null) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: 268435456
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (secstorage-1:null) Is ROOT volume READY (pool already allocated)?: No
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.FirstFitPlanner] (secstorage-1:null) Searching all possible resources under this Zone: 1
2014-04-16 03:22:28,776 DEBUG [cloud.deploy.FirstFitPlanner] (secstorage-1:null) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2014-04-16 03:22:28,779 DEBUG [cloud.deploy.FirstFitPlanner] (secstorage-1:null) No clusters found having a host with enough capacity, returning.
2014-04-16 03:22:28,784 DEBUG [cloud.capacity.CapacityManagerImpl] (secstorage-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: null
2014-04-16 03:22:28,786 WARN [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Exception while trying to start secondary storage vm
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[SecondaryStorageVm|s-316-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
at com.cloud.storage.secondary.SecondaryStorageManagerImpl.startSecStorageVm(SecondaryStorageManagerImpl.java:267)
at com.cloud.storage.secondary.SecondaryStorageManagerImpl.allocCapacity(SecondaryStorageManagerImpl.java:696)
at com.cloud.storage.secondary.SecondaryStorageManagerImpl.expandPool(SecondaryStorageManagerImpl.java:1300)
at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:123)
at com.cloud.secstorage.PremiumSecondaryStorageManagerImpl.scanPool(PremiumSecondaryStorageManagerImpl.java:50)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:104)
at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
2014-04-16 03:22:28,787 INFO [storage.secondary.SecondaryStorageManagerImpl] (secstorage-1:null) Unable to start secondary storage vm for standby capacity, secStorageVm vm Id : 316, will recycle it and start a new one
2014-04-16 03:22:28,787 INFO [cloud.secstorage.PremiumSecondaryStorageManagerImpl] (secstorage-1:null) Primary secondary storage is not even started, wait until next turn
2014-04-16 03:22:28,923 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Zone 1 is ready to launch console proxy
2014-04-16 03:22:28,923 DEBUG [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Expand console proxy standby capacity for zone Zone1
2014-04-16 03:22:28,925 INFO [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Found a stopped console proxy, bring it up to running pool. proxy vm id : 2
2014-04-16 03:22:28,929 DEBUG [cloud.capacity.CapacityManagerImpl] (consoleproxy-1:null) VM state transitted from :Stopped to Starting with event: StartRequestedvm's original host id: null new host id: null host id before state transition: null
2014-04-16 03:22:28,929 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Successfully transitioned to start state for VM[ConsoleProxy|v-2-VM] reservation id = b95325ab-1513-4131-9611-a3be07eb0b4a
2014-04-16 03:22:28,932 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Trying to deploy VM, vm has dcId: 1 and podId: null
2014-04-16 03:22:28,932 DEBUG [cloud.vm.VirtualMachineManagerImpl] (consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2014-04-16 03:22:28,933 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (consoleproxy-1:null) Deploy avoids pods: null, clusters: null, hosts: null
2014-04-16 03:22:28,934 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (consoleproxy-1:null) DeploymentPlanner allocation algorithm: com.cloud.deploy.FirstFitPlanner_EnhancerByCloudStack_9c406fc8@6337c832
2014-04-16 03:22:28,934 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (consoleproxy-1:null) Trying to allocate a host and storage pools from dc:1, pod:null,cluster:null, requested cpu: 500, requested ram: 1073741824
2014-04-16 03:22:28,934 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (consoleproxy-1:null) Is ROOT volume READY (pool already allocated)?: No
2014-04-16 03:22:28,934 DEBUG [cloud.deploy.FirstFitPlanner] (consoleproxy-1:null) Searching all possible resources under this Zone: 1
2014-04-16 03:22:28,935 DEBUG [cloud.deploy.FirstFitPlanner] (consoleproxy-1:null) Listing clusters in order of aggregate capacity, that have (atleast one host with) enough CPU and RAM capacity under this Zone: 1
2014-04-16 03:22:28,936 DEBUG [cloud.deploy.FirstFitPlanner] (consoleproxy-1:null) No clusters found having a host with enough capacity, returning.
2014-04-16 03:22:28,946 DEBUG [cloud.capacity.CapacityManagerImpl] (consoleproxy-1:null) VM state transitted from :Starting to Stopped with event: OperationFailedvm's original host id: null new host id: null host id before state transition: null
2014-04-16 03:22:28,947 WARN [cloud.consoleproxy.ConsoleProxyManagerImpl] (consoleproxy-1:null) Exception while trying to start console proxy
com.cloud.exception.InsufficientServerCapacityException: Unable to create a deployment for VM[ConsoleProxy|v-2-VM]Scope=interface com.cloud.dc.DataCenter; id=1
at com.cloud.vm.VirtualMachineManagerImpl.advanceStart(VirtualMachineManagerImpl.java:841)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:577)
at com.cloud.vm.VirtualMachineManagerImpl.start(VirtualMachineManagerImpl.java:570)
at com.cloud.consoleproxy.ConsoleProxyManagerImpl.startProxy(ConsoleProxyManagerImpl.java:556)
at com.cloud.consoleproxy.ConsoleProxyManagerImpl.allocCapacity(ConsoleProxyManagerImpl.java:928)
at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:1672)
at com.cloud.consoleproxy.ConsoleProxyManagerImpl.expandPool(ConsoleProxyManagerImpl.java:157)
at com.cloud.vm.SystemVmLoadScanner.loadScan(SystemVmLoadScanner.java:111)
at com.cloud.vm.SystemVmLoadScanner.access$100(SystemVmLoadScanner.java:33)
at com.cloud.vm.SystemVmLoadScanner$1.reallyRun(SystemVmLoadScanner.java:81)
at com.cloud.vm.SystemVmLoadScanner$1.run(SystemVmLoadScanner.java:72)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

I tried checking the op_host_capacity table from the db, here it is:

------------------------------------------------------------------------------------------------------------------------------------------------------+

id

host_id

data_center_id

pod_id

cluster_id

used_capacity

reserved_capacity

total_capacity

capacity_type

capacity_state

update_time

created

------------------------------------------------------------------------------------------------------------------------------------------------------+

1	NULL	1	NULL	NULL	0	0	50	8	Enabled	2014-03-26 20:15:28	2014-03-26 20:10:28
2	NULL	1	1	NULL	1	0	253	5	Enabled	2014-03-26 20:20:28	2014-03-26 20:10:28
3	1	1	1	1	0	0	148180032	0	Enabled	2014-04-16 00:24:01	2014-03-26 20:18:01
4	1	1	1	1	0	0	2892	1	Enabled	2014-03-26 20:18:01	2014-03-26 20:18:01
5	2	1	1	1	0	0	37004247040	3	Enabled	2014-03-26 20:19:11	2014-03-26 20:19:11

------------------------------------------------------------------------------------------------------------------------------------------------------+
5 rows in set (0.00 sec)

I have not used any of my capacity yet I still cannot start my system VM. Please help. Thanks