[CLOUDSTACK-4402] [deleteStoragePool] There is no way to delete Primary storage if the last host with which it was associated is already removed.

• Type: Bug
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.2.0
• Fix Version/s: 4.3.0
• Component/s: Management Server
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: commit id # 3c40e8bb3f6278f78c24c6317d513bd5ad599944
• Assignee: Sanjay Tripathi
• Reporter: Venkata Swamy
• Created: 19/Aug/13 13:48
• Updated: 02/Jan/14 11:03
• Resolved: 27/Dec/13 12:56

Description

Steps to reproduce :

1. Have at least 1 advanced zone with XenServer Cluster
2. Doesn't actually matter whether VMs are deployed or not.

In my case the exact environment is as mentioned below

• 1 advanced zone
• 1 KVM cluster with 1 host
• 1 XenServer Cluster with 1 host
• All systemVMs are running on KVM host

3. place the XenServer into maintenance mode

2013-08-19 19:05:31,905 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===START=== 10.252.192.25 – GET command=deleteHost&id=f498aa17-ed6c-4ca1-92e8-ef32c70c9290&forced=false&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919346034
2013-08-19 19:05:31,919 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-22:null) Dispatching resource state event DELETE_HOST to NetworkUsageManagerImpl_EnhancerByCloudStack_aa8b2ad3
2013-08-19 19:05:31,919 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-22:null) Dispatching resource state event DELETE_HOST to KvmServerDiscoverer_EnhancerByCloudStack_1647b3bf
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-22:null) Dispatching resource state event DELETE_HOST to OvmDiscoverer_EnhancerByCloudStack_829e4aea
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-22:null) Dispatching resource state event DELETE_HOST to VmwareServerDiscoverer_EnhancerByCloudStack_6ca85485
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-22:null) Dispatching resource state event DELETE_HOST to PremiumSecondaryStorageManagerImpl_EnhancerByCloudStack_4b3fda74
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-22:null) Dispatching resource state event DELETE_HOST to XcpServerDiscoverer_EnhancerByCloudStack_1934d343
2013-08-19 19:05:31,920 DEBUG [cloud.resource.ResourceManagerImpl] (catalina-exec-22:null) Deleting Host: 9 Guid:d1f2d15f-17a5-4495-8f33-e8f5a5418413
2013-08-19 19:05:31,924 DEBUG [dc.dao.DataCenterIpAddressDaoImpl] (catalina-exec-22:null) Releasing ip address: 10.147.40.31 data center 1
2013-08-19 19:05:31,926 INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Host 9 is disconnecting with event Remove
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) The next status of agent 9is Removed, current status is Up
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Deregistering link for 9 with state Removed
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Remove Agent : 9
2013-08-19 19:05:31,928 DEBUG [agent.manager.DirectAgentAttache] (AgentTaskPool-9:null) Processing disconnect 9
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.hypervisor.xen.discoverer.XcpServerDiscoverer_EnhancerByCloudStack_1934d343
2013-08-19 19:05:31,928 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.deploy.DeploymentPlanningManagerImpl_EnhancerByCloudStack_dff95db9
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.network.NetworkManagerImpl_EnhancerByCloudStack_f18a6f4c
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.storage.secondary.SecondaryStorageListener
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.hypervisor.vmware.manager.VmwareManagerImpl_EnhancerByCloudStack_88ff6e46
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.network.security.SecurityGroupListener
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.storage.listener.StoragePoolMonitor
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.vm.ClusteredVirtualMachineManagerImpl_EnhancerByCloudStack_1e4b2050
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.storage.LocalStoragePoolListener
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.network.router.VirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_b7bc8cf1
2013-08-19 19:05:31,929 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.network.SshKeysDistriMonitor
2013-08-19 19:05:31,930 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.network.router.VpcVirtualNetworkApplianceManagerImpl_EnhancerByCloudStack_32a05514
2013-08-19 19:05:31,931 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.storage.upload.UploadListener
2013-08-19 19:05:31,931 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.storage.download.DownloadListener
2013-08-19 19:05:31,931 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.agent.manager.AgentMonitor
2013-08-19 19:05:31,932 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.capacity.StorageCapacityListener
2013-08-19 19:05:31,932 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.capacity.ComputeCapacityListener
2013-08-19 19:05:31,932 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.network.NetworkUsageManagerImpl$DirectNetworkStatsListener
2013-08-19 19:05:31,932 DEBUG [cloud.network.NetworkUsageManagerImpl] (AgentTaskPool-9:null) Disconnected called on 9 with status Removed
2013-08-19 19:05:31,932 DEBUG [agent.manager.AgentManagerImpl] (AgentTaskPool-9:null) Sending Disconnect to listener: com.cloud.consoleproxy.ConsoleProxyListener
2013-08-19 19:05:31,932 DEBUG [cloud.host.Status] (AgentTaskPool-9:null) Transition:[Resource state = Maintenance, Agent event = Remove, Host id = 9, name = Rack1Pod1Host31]
2013-08-19 19:05:31,950 DEBUG [cloud.resource.ResourceState] (catalina-exec-22:null) Resource state update: [id = 9; name = Rack1Pod1Host31; old state = Maintenance; event = DeleteHost; new state = Disabled]
2013-08-19 19:05:31,955 DEBUG [cloud.host.Status] (AgentTaskPool-9:null) Agent status update: [id = 9; name = Rack1Pod1Host31; old status = Up; event = Remove; new status = Removed; old update count = 2; new update count = 3]
2013-08-19 19:05:31,955 DEBUG [agent.manager.ClusteredAgentManagerImpl] (AgentTaskPool-9:null) Notifying other nodes of to disconnect
2013-08-19 19:05:31,961 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===END=== 10.252.192.25 – GET command=deleteHost&id=f498aa17-ed6c-4ca1-92e8-ef32c70c9290&forced=false&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919346034

4. remove the host first

5. Go the primary storage that is associated with Xenserver and place that into maintenance mode

2013-08-19 19:06:44,458 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null) ===END=== 10.252.192.25 – GET command=enableStorageMaintenance&id=a1a1a150-6709-3d81-a414-4cee93e62168&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919418572
2013-08-19 19:06:44,461 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-63:job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]) Executing org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd for job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]
2013-08-19 19:06:44,481 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-63:job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]) Complete async job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ], jobStatus: 1, resultCode: 0, result: org.apache.cloudstack.api.response.StoragePoolResponse@5de2af25
2013-08-19 19:06:44,485 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-63:job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]) Done executing org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd for job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ]
2013-08-19 19:06:47,504 DEBUG [cloud.api.ApiServlet] (catalina-exec-20:null) ===START=== 10.252.192.25 – GET command=queryAsyncJobResult&jobId=0b3ebe91-0101-4ba2-bb36-2327b9e7566f&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919421729
2013-08-19 19:06:47,512 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-20:null) Async job-62 = [ 0b3ebe91-0101-4ba2-bb36-2327b9e7566f ] completed

6. Now try to remove the host with forced=false

commanddeleteStoragePool&forced=false&id=a1a1a150-6709-3d81-a414-4cee93e62168responsejsonsessionkeyzJzyl0NdZA0QlMd8kSwWbynDxBI=

Observations :-

It failed to delete with an error saying "Failed to delete storage pool on host"

(ii) tried to delete it with forced=true and that as well failed with the same error.

2013-08-19 19:06:55,677 ERROR [cloud.api.ApiServer] (catalina-exec-22:null) unhandled exception executing api command: deleteStoragePool
com.cloud.utils.exception.CloudRuntimeException: Failed to delete storage pool on host
at org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.deleteDataStore(CloudStackPrimaryDataStoreLifeCycleImpl.java:478)
at com.cloud.storage.StorageManagerImpl.deletePool(StorageManagerImpl.java:937)
at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at org.apache.cloudstack.api.command.admin.storage.DeletePoolCmd.execute(DeletePoolCmd.java:78)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
at com.cloud.api.ApiServer.queueCommand(ApiServer.java:514)
at com.cloud.api.ApiServer.handleRequest(ApiServer.java:372)
at com.cloud.api.ApiServlet.processRequest(ApiServlet.java:305)
at com.cloud.api.ApiServlet.doGet(ApiServlet.java:66)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:617)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
at org.apache.coyote.http11.Http11NioProcessor.process(Http11NioProcessor.java:889)
at org.apache.coyote.http11.Http11NioProtocol$Http11ConnectionHandler.process(Http11NioProtocol.java:721)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:2268)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:679)
2013-08-19 19:06:55,679 DEBUG [cloud.api.ApiServlet] (catalina-exec-22:null) ===END=== 10.252.192.25 – GET command=deleteStoragePool&id=a1a1a150-6709-3d81-a414-4cee93e62168&forced=false&response=json&sessionkey=zJzyl0NdZA0QlMd8kSwWbynDxBI%3D&_=1376919429782

Attaching all the required logs along with db dump to the bug.

Expected Result:

Either we should not let admin remove the hypervisor host while it has primary storages associated and mounted on it

OR

(ii) we should let the deleteStoragePool with forced=true to delete the pool forcefully.

Issue Links

• is duplicated by: CLOUDSTACK-4697 Not able to delete Primary storage when there are no hosts in the cluster.