[CLOUDSTACK-2780] primary storage is already mounted but not found in pool-list

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 4.0.1
• Security Level: Public (Anyone can view this level - this is the default.)
• Assignee: Wei Zhou
• Reporter: Wei Zhou
• Created: 31/May/13 06:25
• Updated: 18/Sep/13 11:51
• Resolved: 18/Sep/13 11:50

Description

This issue appeared on our testing enviroment with cloudstack 4.0.1 (Ubuntu 12.04) , and product platform as well.
(only on few hosts, most of hosts work well)

We got the message "Unable to create volume" when deploy a VM.

(1) In management-server.log

2013-05-30 14:36:20,090 DEBUG [agent.transport.Request] (AgentManager-Handler-11:null) Seq 3-2021457981: Processing: { Ans: , MgmtId: 345051509349, via: 3, Ver: v1, Flags: 110, [{"storage.CreateAnswer":{"requestTemplateReload":false,"result":false,"details":"Exception: com.cloud.utils.exception.CloudRuntimeException\nMessage: org.libvirt.LibvirtException: Storage pool not found: no pool with matching uuid\nStack: com.cloud.utils.exception.CloudRuntimeException: org.libvirt.LibvirtException: Storage pool not found: no pool with matching uuid\n\tat com.cloud.hypervisor.kvm.storage.LibvirtStorageAdaptor.getStoragePool(LibvirtStorageAdaptor.java:394)\n\tat com.cloud.hypervisor.kvm.storage.KVMStoragePoolManager.getStoragePool(KVMStoragePoolManager.java:48)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.execute(LibvirtComputingResource.java:1212)\n\tat com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1031)\n\tat com.cloud.agent.Agent.processRequest(Agent.java:518)\n\tat com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:831)\n\tat com.cloud.utils.nio.Task.run(Task.java:83)\n\tat java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)\n\tat java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)\n\tat java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-05-30 14:36:20,090 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-11:null) Seq 3-2021457981: No more commands found
2013-05-30 14:36:20,090 DEBUG [agent.transport.Request] (Job-Executor-5:job-2563) Seq 3-2021457981: Received: { Ans: , MgmtId: 345051509349, via: 3, Ver: v1, Flags: 110,

{ CreateAnswer }

}
2013-05-30 14:36:20,090 DEBUG [cloud.storage.StorageManagerImpl] (Job-Executor-5:job-2563) Unable to create volume Vol[750|vm=550|ROOT]
2013-05-30 14:36:20,095 INFO [cloud.vm.VirtualMachineManagerImpl] (Job-Executor-5:job-2563) Unable to contact resource.
com.cloud.exception.StorageUnavailableException: Resource [StoragePool:200] is unreachable: Unable to create Vol[750|vm=550|ROOT]

(2) In the host Primary storage is already mounted (just an example):

[root@cs-kvm015 /]# df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/vg0-root 461410320 2592404 435379580 1% /
tmpfs 8160712 0 8160712 0% /dev/shm
/dev/sda1 198337 52765 135332 29% /boot
/dev/mapper/vg0-tmp 2064208 68648 1890704 4% /tmp
192.168.103.1:/storage/cs4x-primary
7811748864 466487296 7345261568 6% /mnt/7b88eced-fc54-3c9f-808a-a697d6be667c

But not in libvirt pool list

[root@cs-kvm015 ~]# virsh pool-list
Name State Autostart
-----------------------------------------
513ae20a-61f9-4f7a-b485-23ddb29e59ae active no
7b88eced-fc54-3c9f-808a-a697d6be667c active no (should have this line, but have not)

(3) Workaround:

Enable Maintenance Mode
umount /mnt/7b88eced-fc54-3c9f-808a-a697d6be667c
service cloud-agent restart

Issue Links

• is duplicated by: CLOUDSTACK-2893 The Agent attempts to re-create a already existing Libvirt Storage pool when creating a volume
• relates to: CLOUDSTACK-2729 [Automation] Libvirt failed find primary storage and VM deployment failed
• relates to: CLOUDSTACK-3565 Restarting libvirtd service leading to destroy storage pool
• relates to: CLOUDSTACK-4459 [Automation] Vm deployment failed in KVM with missing storage pool error
• relates to: CLOUDSTACK-2893 The Agent attempts to re-create a already existing Libvirt Storage pool when creating a volume