[CLOUDSTACK-9864] cleanup stale worker VMs after job expiry time

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Component/s: VMware
• Security Level: Public (Anyone can view this level - this is the default.)
• Labels: vmware vsphere workers
• Assignee: Daan
• Reporter: Daan
• Created: 06/Apr/17 08:27
• Updated: 02/May/17 15:37
• Resolved: 02/May/17 15:37

Description

In the present code cleaning worker vms after a timeout is disabled, with the documented reason that there is no API to query for related tasks in vcenter. ACS has an expiry time for jobs and a cancel time for jobs.

• Jobs that take longer then the expiry time will have their results be be neglected.
• Jobs that are cancelled are forcibly removed after the cancellation expity time.
  Any worker remaining after expiry+cancellation will surely be stale and can be removed.

As some administrators may not want this behaviour there will be a setting which by default is false that will guard against cleaning stale worker VMs.
Stale worker VMs will be cleaned after 2 * (expiry-time + cancellation-time) as a safe margin.

related settings:
job.expire.minutes: 1440
job.cancel.threshold.minutes: 60
vmware.clean.old.worker.vms: false (new)

Issue Links

• links to: GitHub Pull Request #2030