[CLOUDSTACK-5401] VM migration during host maintenance fails if pool.storage.capacity.disablethreshold is lowered

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.2.1, 4.3.0
• Fix Version/s: 4.3.0
• Component/s: Management Server
• Security Level: Public (Anyone can view this level - this is the default.)
• Assignee: Prachi Damle
• Reporter: Prachi Damle
• Created: 06/Dec/13 09:40
• Updated: 26/Feb/14 07:43
• Resolved: 05/Feb/14 00:35

Description

1. Create a 2 host XS 6.0.2 cluster (H1 and H2)
2. Create 6 or more VMs such that they get created in H1
3. Lower pool.storage.capacity.disablethreshold to 0.1 (default is 0.85)
4. Put H1 into maintenance. Some or all guest VMs fail to migrate to H2

2013-11-25 15:41:12,098 DEBUG [cloud.deploy.FirstFitPlanner] (HA-Worker-3:work-28) The specified cluster is in avoid set, returning.
2013-11-25 15:41:12,098 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-28) Unable to find destination for migrating the vm VM[User|z1V6]
2013-11-25 15:41:12,098 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-28) Insufficient capacity for migrating a VM.
2013-11-25 15:41:12,099 DEBUG [cloud.resource.ResourceManagerImpl] (HA-Worker-3:work-28) No next resource state for host 5 while current state is ErrorInMaintenance with event UnableToMigrate
com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =ErrorInMaintenance event =UnableToMigrate
at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
at com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
at com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:610)
at com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
2013-11-25 15:41:12,100 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-28) Rescheduling HAWork[28-Migration-9-Running-Migrating] to try again at Mon Nov 25 15:43:14 PST 2013
2013-11-25 15:41:12,100 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HA-Worker-4:work-29) Checking suitable pools for volume (Id, Type): (13,ROOT)
2013-11-25 15:41:12,100 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HA-Worker-4:work-29) We need to allocate new storagepool for this volume
2013-11-25 15:41:12,102 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HA-Worker-4:work-29) Calling StoragePoolAllocators to find suitable pools
2013-11-25 15:41:12,103 DEBUG [storage.allocator.LocalStoragePoolAllocator] (HA-Worker-4:work-29) LocalStoragePoolAllocator trying to find storage pool to fit the vm
2013-11-25 15:41:12,103 DEBUG [storage.allocator.ClusterScopeStoragePoolAllocator] (HA-Worker-4:work-29) ClusterScopeStoragePoolAllocator looking for storage pool
2013-11-25 15:41:12,103 DEBUG [storage.allocator.ClusterScopeStoragePoolAllocator] (HA-Worker-4:work-29) Looking for pools in dc: 1 pod:1 cluster:1
2013-11-25 15:41:12,107 DEBUG [storage.allocator.AbstractStoragePoolAllocator] (HA-Worker-4:work-29) Checking if storage pool is suitable, name: null ,poolId: 200
2013-11-25 15:41:12,111 DEBUG [cloud.storage.StorageManagerImpl] (HA-Worker-4:work-29) Checking pool 200 for storage, totalSize: 11810778316800, usedBytes: 9755417411584, usedPct: 0.8259758290194649, disable threshold: 0.1
2013-11-25 15:41:12,111 DEBUG [cloud.storage.StorageManagerImpl] (HA-Worker-4:work-29) Insufficient space on pool: 200 since its usage percentage: 0.8259758290194649 has crossed the pool.storage.capacity.disablethreshold: 0.1
2013-11-25 15:41:12,111 DEBUG [storage.allocator.ClusterScopeStoragePoolAllocator] (HA-Worker-4:work-29) ClusterScopeStoragePoolAllocator returning 0 suitable storage pools
2013-11-25 15:41:12,111 DEBUG [storage.allocator.ZoneWideStoragePoolAllocator] (HA-Worker-4:work-29) ZoneWideStoragePoolAllocator to find storage pool
2013-11-25 15:41:12,113 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HA-Worker-4:work-29) No suitable pools found for volume: Vol[13|vm=12|ROOT] under cluster: 1
2013-11-25 15:41:12,113 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HA-Worker-4:work-29) No suitable pools found
2013-11-25 15:41:12,113 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HA-Worker-4:work-29) No suitable storagePools found under this Cluster: 1
2013-11-25 15:41:12,117 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HA-Worker-4:work-29) Could not find suitable Deployment Destination for this VM under any clusters, returning.

-2013-11-25 15:32:09,784 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-3:work-27) Unable to find destination for migrating the vm VM[User|z1V5]
2013-11-25 15:32:09,784 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-27) Insufficient capacity for migrating a VM.
2013-11-25 15:32:09,784 DEBUG [cloud.deploy.FirstFitPlanner] (HA-Worker-1:work-29) The specified cluster is in avoid set, returning.
2013-11-25 15:32:09,784 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-1:work-29) Unable to find destination for migrating the vm VM[SecondaryStorageVm|s-12-VM]
2013-11-25 15:32:09,784 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-29) Insufficient capacity for migrating a VM.
2013-11-25 15:32:09,786 DEBUG [cloud.deploy.DeploymentPlanningManagerImpl] (HA-Worker-4:work-28) Could not find suitable Deployment Destination for this VM under any clusters, returning.
2013-11-25 15:32:09,786 DEBUG [cloud.deploy.FirstFitPlanner] (HA-Worker-4:work-28) Searching resources only under specified Cluster: 1
2013-11-25 15:32:09,788 DEBUG [cloud.resource.ResourceManagerImpl] (HA-Worker-3:work-27) No next resource state for host 5 while current state is ErrorInMaintenance with event UnableToMigrate
com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =ErrorInMaintenance event =UnableToMigrate
at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
at com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
at com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:610)
at com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
2013-11-25 15:32:09,789 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-3:work-27) Rescheduling HAWork[27-Migration-8-Running-Migrating] to try again at Mon Nov 25 15:34:11 PST 2013
2013-11-25 15:32:09,790 DEBUG [cloud.deploy.FirstFitPlanner] (HA-Worker-4:work-28) The specified cluster is in avoid set, returning.
2013-11-25 15:32:09,790 DEBUG [cloud.vm.VirtualMachineManagerImpl] (HA-Worker-4:work-28) Unable to find destination for migrating the vm VM[User|z1V6]
2013-11-25 15:32:09,790 WARN [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-28) Insufficient capacity for migrating a VM.
2013-11-25 15:32:09,790 DEBUG [cloud.resource.ResourceManagerImpl] (HA-Worker-1:work-29) No next resource state for host 5 while current state is ErrorInMaintenance with event UnableToMigrate
com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =ErrorInMaintenance event =UnableToMigrate
at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
at com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
at com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:610)
at com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
2013-11-25 15:32:09,790 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-1:work-29) Rescheduling HAWork[29-Migration-12-Running-Migrating] to try again at Mon Nov 25 15:34:11 PST 2013
2013-11-25 15:32:09,793 DEBUG [cloud.resource.ResourceManagerImpl] (HA-Worker-4:work-28) No next resource state for host 5 while current state is ErrorInMaintenance with event UnableToMigrate
com.cloud.utils.fsm.NoTransitionException: No next resource state found for current state =ErrorInMaintenance event =UnableToMigrate
at com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178)
at com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313)
at com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:610)
at com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858)
2013-11-25 15:32:09,793 INFO [cloud.ha.HighAvailabilityManagerImpl] (HA-Worker-4:work-28) Rescheduling HAWork[28-Migration-9-Running-Migrating] to try again at Mon Nov 25 15:34:11 PST 2013
2013-11-25 15:32:15,047 DEBUG [cloud.server.StatsCollector] (StatsCollector-2:null) HostStatsCollector is running...
2013-11-25 15:32:15,059 DEBUG [agent.manager.DirectAgentAttache] (DirectAgent-85:null) Seq 1-2115240044: Executing request
2013-11-25 15:32:15,162 DEBUG [agent.manager.AgentManagerImpl] (AgentManager-Handler-6:null) SeqA 3-1080: Processing Seq 3-1080: { Cmd , MgmtId: -1, via: 3, Ver: v1, Flags: 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand
":

{"_proxyVmId":2,"_loadInfo":" {\n \"connections\": []\n}