Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-5678

Cold Storage migration is failing

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Closed
    • Blocker
    • Resolution: Fixed
    • 4.3.0
    • 4.4.0
    • Management Server
    • Security Level: Public (Anyone can view this level - this is the default.)
    • hyperv , 4.3

    Description

      Steps to Reproduce:
      ================
      1.Bring up CS in advanced zone with 2 hosts in a hyper-v cluster using CIFS for both primary and secondary storage
      2.Deploy one or two guest vms using default cent os template.
      3.Add one more primary storage (say primary2) to the cluster
      4.Stop one of the vms deployed at step2
      5.Migrate the storage from source primary to new primary storage primary2

      Expected Result:
      =============
      Storage migration should succeed when the vm is in stopped state

      Actual Result:
      ===========
      013-12-30 17:21:03,163 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-9672ce4b) ===START=== 10.144.7.20 – GET command=migrateVirtualMachine&storageid=d65ffa8c-fdd7-360a-b729-c0d9ba5df050&virtualmachineid=9226ccd6-b409-4452-821f-5b33a24137a5&response=json&sessionkey=lIed4xLaMsDL4yCyAJYyUK0pAJk%3D&_=1388403965445
      2013-12-30 17:21:03,194 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (catalina-exec-6:ctx-9672ce4b ctx-5d353029) submit async job-129, details: AsyncJobVO {id:129, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:

      {"response":"json","sessionkey":"lIed4xLaMsDL4yCyAJYyUK0pAJk\u003d","virtualmachineid":"9226ccd6-b409-4452-821f-5b33a24137a5","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"d65ffa8c-fdd7-360a-b729-c0d9ba5df050","httpmethod":"GET","_":"1388403965445","ctxAccountId":"2","ctxStartEventId":"274"}

      , cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
      2013-12-30 17:21:03,194 DEBUG [c.c.a.ApiServlet] (catalina-exec-6:ctx-9672ce4b ctx-5d353029) ===END=== 10.144.7.20 – GET command=migrateVirtualMachine&storageid=d65ffa8c-fdd7-360a-b729-c0d9ba5df050&virtualmachineid=9226ccd6-b409-4452-821f-5b33a24137a5&response=json&sessionkey=lIed4xLaMsDL4yCyAJYyUK0pAJk%3D&_=1388403965445
      2013-12-30 17:21:03,196 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Job-Executor-48:ctx-46f6bd2b) Add job-129 into job monitoring
      2013-12-30 17:21:03,196 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-48:ctx-46f6bd2b) Executing AsyncJobVO {id:129, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:

      {"response":"json","sessionkey":"lIed4xLaMsDL4yCyAJYyUK0pAJk\u003d","virtualmachineid":"9226ccd6-b409-4452-821f-5b33a24137a5","cmdEventType":"VM.MIGRATE","ctxUserId":"2","storageid":"d65ffa8c-fdd7-360a-b729-c0d9ba5df050","httpmethod":"GET","_":"1388403965445","ctxAccountId":"2","ctxStartEventId":"274"}

      , cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
      2013-12-30 17:21:03,212 DEBUG [c.c.u.d.T.Transaction] (Job-Executor-48:ctx-46f6bd2b ctx-5d353029) Rolling back the transaction: Time = 4 Name = Job-Executor-48; called by -TransactionLegacy.rollback:896-TransactionLegacy.removeUpTo:839-TransactionLegacy.close:663-Transaction.execute:41-Transaction.execute:46-VirtualMachineManagerImpl.migrateVmStorageThroughJobQueue:4475-VirtualMachineManagerImpl.storageMigration:1558-UserVmManagerImpl.vmStorageMigration:4069-NativeMethodAccessorImpl.invoke0:-2-NativeMethodAccessorImpl.invoke:57-DelegatingMethodAccessorImpl.invoke:43-Method.invoke:616
      2013-12-30 17:21:03,213 ERROR [c.c.a.ApiAsyncJobDispatcher] (Job-Executor-48:ctx-46f6bd2b) Unexpected exception while executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd
      com.cloud.utils.exception.CloudRuntimeException: Unable to serialize: com.cloud.vm.VmWorkStorageMigration@40df9e44
      at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.toObjectSerializedString(JobSerializerHelper.java:118)
      at com.cloud.vm.VmWorkSerializer.serialize(VmWorkSerializer.java:63)
      at com.cloud.vm.VirtualMachineManagerImpl$9.doInTransactionWithoutResult(VirtualMachineManagerImpl.java:4504)
      at com.cloud.utils.db.TransactionCallbackNoReturn.doInTransaction(TransactionCallbackNoReturn.java:25)
      at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction.java:49)
      at com.cloud.utils.db.Transaction.execute(Transaction.java:37)
      at com.cloud.utils.db.Transaction.execute(Transaction.java:46)
      at com.cloud.vm.VirtualMachineManagerImpl.migrateVmStorageThroughJobQueue(VirtualMachineManagerImpl.java:4475)
      at com.cloud.vm.VirtualMachineManagerImpl.storageMigration(VirtualMachineManagerImpl.java:1558)
      at com.cloud.vm.UserVmManagerImpl.vmStorageMigration(UserVmManagerImpl.java:4069)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)

      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
      at $Proxy169.vmStorageMigration(Unknown Source)
      at org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd.execute(MigrateVMCmd.java:150)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:161)
      at com.cloud.api.ApiAsyncJobDispatcher.runJobInContext(ApiAsyncJobDispatcher.java:109)
      at com.cloud.api.ApiAsyncJobDispatcher$1.run(ApiAsyncJobDispatcher.java:66)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:63)
      at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
      at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
      at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)
      Caused by: java.io.NotSerializableException: org.apache.cloudstack.storage.datastore.db.PrimaryDataStoreDaoImpl
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1180)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)
      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1528)

      at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1493)
      at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1416)
      at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1174)
      at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:346)
      at org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.toObjectSerializedString(JobSerializerHelper.java:112)
      ... 40 more
      2013-12-30 17:21:03,222 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-48:ctx-46f6bd2b) Complete async job-129, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/

      {"uuidList":[],"errorcode":530,"errortext":"Unable to serialize: com.cloud.vm.VmWorkStorageMigration@40df9e44"}

      2013-12-30 17:21:03,226 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Job-Executor-48:ctx-46f6bd2b) Done executing org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-129

      Workaround :
      =========================
      Add a new entry in cloud.configurations table with

      vm.job.enabled set to "false"

      restart management server. After this Live VM migration works fine.

      Attachments

        Activity

          People

            kelveny Kelven Yang
            abhinavr Abhinav Roy
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: