[CLOUDSTACK-9472] Taking snapshot on a large VMware volume times out

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 4.9.0
• Component/s: Snapshot, VMware
• Security Level: Public (Anyone can view this level - this is the default.)
• Environment: * VMware vCenter 5.5U3/ESXi 5.0 U1 ; * ACS 4.9 running on CentOS6.4
• Assignee: Unassigned
• Reporter: Boris Stoyanov
• Created: 25/Aug/16 08:02
• Updated: 06/Feb/17 04:18
• Resolved: 06/Feb/17 04:18

Description

Description:
Taking snapshot on large volume (more than 100GB) will fail on VMware Cluster.

Setup:
1 VMware cluster with 2 Hosts.

Executed Steps:

1. Set global variables
   • backup.snapshot.wait to 6000
   • copy.volume.wait to 6000
   • wait = 3600
   • vmware.vcenter.session.timout - 6000
2. Restarted management server.
3. Created a large volume – 350GB
4. Deployed a VM in the VMware Cluster
5. Attached the Volume to the VM.
6. Start a Volume snapshot.

Result: A timeout occurred when creating the snapshot after ~60mins.
This is the only error message we managed to find in the log:

2016-08-19 04:18:52,559 INFO  [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-54fa6dd0) (logid:b8357ae4) End cleanup expired async-jobs
2016-08-19 04:18:52,598 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-fbf1081e job-356) (logid:55a8f0e8) Complete async job-356, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed to create snapshot due to an internal error creating snapshot for volume 102"}
2016-08-19 04:18:52,601 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-1:ctx-fbf1081e job-356) (logid:55a8f0e8) Publish async job-356 complete on message bus

Please see full log in the attached file.