Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-10305

Rare race condition in KVM migration

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • None
    • 4.11.1.0
    • None
    • Security Level: Public (Anyone can view this level - this is the default.)
    • None

    Description

      There is a race condition in the monitoring of the migration process on KVM. If the monitor wakes up in the tight window after the migration succeeds, but before the migration thread terminates, the monitor will get a LibvirtException “Domain not found: no domain with matching uuid” when checking on the migration status. This in turn causes CloudStack to sync the VM state to stop, in which it issues a defensive StopCommand to ensure it is correctly synced.

      This is issue is reported under these conditions:

      • VM migration has physically finished
      • Migration thread executor is still not marked as terminated

      From the migration monitoring process, as VM has been migrated, that domain cannot be found (we should get the destination domain from the migration thread later) and the LibvirtException is thrown: LibvirtException “Domain not found: no domain with matching uuid”.
      So basically the idea is just logging the issue but not suspending the VM as the migration has already been performed, but the migration thread is not marked as finished. Once the migrarion thread is marked as finished, the migration monitoring process (while loop) ends and the migration command wrapper would continue its execution.
      Regarding your last question, we won't need to suspend the VM in this case as the migration has already been performed. VM suspension will take place whenever the migration is in progress and those conditions are met

      Attachments

        Issue Links

          Activity

            People

              nicolas.vazquez Nicolas Vazquez
              nicolas.vazquez Nicolas Vazquez
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: