Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-10326

Prevent hosts fall into Maintenance when there are running VMs on it

    XMLWordPrintableJSON

Details

    • Bug
    • Status: In Progress
    • Major
    • Resolution: Unresolved
    • 4.11.0.0
    • 4.11.1.0
    • None
    • Security Level: Public (Anyone can view this level - this is the default.)
    • None

    Description

      This issue was discovered, fixed and tested on KVM, but applies for every hypervisor.

      Background

      When enabling maintenance mode in a host, host state is put into 'PrepareForMaintenance' and running VMs are migrated into another host. After every VM is migrated, host goes to 'Maintenance' state.

      Checks are performed on ResourceManagerImpl.checkAndMaintan() method:

      • List VMs with host_id = HOST_ID
      • List VMs with last_host_id = HOST_ID and state=Migrating

      When both queries are empty, then the host can be put into Maintenance.

      When a VM is being migrated to DEST_HOST, its host_id column is set to DEST_HOST, last_host_id = ORIGIN_HOST and state = Migrating. If then migration fails, host_id = last_host_id = ORIGIN_HOST 

      Issue

      This sequence:

      • Enable maintenance mode on ORIGIN_HOST
      • VMs start being migrated to a host, say DEST_HOST
      • checkAndMaintain() starts:
        • First check passes (no VM with host_id = ORIGIN_HOST_ID as those are being migrated)
        • Before the second check, one or more migrations fail
        • Second check passes, however there are VMs running on the host as migrations have failed.
      • Host goes into Maintenance state.

      Screenshots attached, query executed on each case:

      select id, name, instance_name, state, host_id, last_host_id from vm_instance;

      Attachments

        1. CLOUDSTACK-10326-MigrationFailed.png
          13 kB
          Nicolas Vazquez
        2. CLOUDSTACK-10326-Migrating.png
          13 kB
          Nicolas Vazquez
        3. CLOUDSTACK-10326-InitialState.png
          13 kB
          Nicolas Vazquez
        4. CLOUDSTACK-10326-Debug.png
          98 kB
          Nicolas Vazquez

        Issue Links

          Activity

            People

              nicolas.vazquez Nicolas Vazquez
              nicolas.vazquez Nicolas Vazquez
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: