Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-4734

Creating snapshot from ROOT volume fails with error message - "Failed to create snapshot due to an internal error creating snapshot for volume 14”

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 4.2.1
    • 4.3.0
    • Doc, Management Server
    • Security Level: Public (Anyone can view this level - this is the default.)
    • None
    • Build from 4.2-forward

    Description

      Set up:

      3.0.7 Advanced Zone setup with 2 Zones having Vmware EXSI 5.0 clusters on Vcenter 5.0 .
      Upgraded to 4.2.0.

      Tried to take snapshot for ROOT volume of 2 existing Vms from 2 different zones.
      Snapshots got created successfully.

      After few hours , when I tried to create snapshot from ROOT volume for a VM, I was not able to create snapshots from ROOT volume.
      I get the following error – “Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume
      14”

      Following is the error message seen on management server logs:

      2013-09-21 06:56:13,573 DEBUG [agent.transport.Request] (Job-Executor-36:job-88 = [ c7cbee54-930d-4020-ae02-4710806
      bdc00 ]) Seq 9-1758069619: Sending { Cmd , MgmtId: 161197867246747, via: 9, Ver: v1, Flags: 100111, [{"org.apache.
      cloudstack.storage.command.CopyCommand":{"srcTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{"path":"9e8a
      10cf-7cbc-4bef-a57b-563edb6b62ed","volume":{"uuid":"05a6bdfd-e6a5-4ae5-9dab-d06ee40776e9","volumeType":"ROOT","data
      Store":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"61e1621f-a1a8-336d-b18d-050586f19b09","id":2
      00,"poolType":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/sangeetha/campofinalga/primary","por
      t":2049}},"name":"ROOT-14","size":2147483648,"path":"ROOT-14-14","volumeId":14,"vmName":"i-3-14-VM","accountId":3,"
      format":"OVA","id":14,"hypervisorType":"VMware"},"dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO"
      :{"uuid":"61e1621f-a1a8-336d-b18d-050586f19b09","id":200,"poolType":"NetworkFilesystem","host":"10.223.110.232","pa
      th":"/export/home/sangeetha/campofinalga/primary","port":2049}},"vmName":"i-3-14-VM","name":"sangee-tier1-4_ROOT-14
      _20130921135613","hypervisorType":"VMware","id":5}},"destTO":{"org.apache.cloudstack.storage.to.SnapshotObjectTO":{
      "path":"snapshots/3/14","volume":{"uuid":"05a6bdfd-e6a5-4ae5-9dab-d06ee40776e9","volumeType":"ROOT","dataStore":{"o
      rg.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"61e1621f-a1a8-336d-b18d-050586f19b09","id":200,"poolTy
      pe":"NetworkFilesystem","host":"10.223.110.232","path":"/export/home/sangeetha/campofinalga/primary","port":2049}},
      "name":"ROOT-14","size":2147483648,"path":"ROOT-14-14","volumeId":14,"vmName":"i-3-14-VM","accountId":3,"format":"O
      VA","id":14,"hypervisorType":"VMware"},"dataStore":{"com.cloud.agent.api.to.NfsTO":{"_url":"nfs://10.223.110.232/ex
      port/home/sangeetha/campofinalga/secondary","_role":"Image"}},"vmName":"i-3-14-VM","name":"sangee-tier1-4_ROOT-14_2
      0130921135613","hypervisorType":"VMware","id":5}},"executeInSequence":true,"wait":21600}}] }

      2013-09-21 06:56:18,263 DEBUG [agent.transport.Request] (AgentManager-Handler-14:null) Seq 9-1758069619: Processing
      : { Ans: , MgmtId: 161197867246747, via: 9, Ver: v1, Flags: 110, [{"org.apache.cloudstack.storage.command.CopyCmdA
      nswer":{"result":false,"details":"backup snapshot exception: Exception: javax.xml.ws.soap.SOAPFaultException\nMessa
      ge: \nRequired property obj is missing from data object of type ObjectSpec\n\nwhile parsing serialized DataObject o
      f type vmodl.query.PropertyCollector.ObjectSpec\nat line 1, column 305\n\nwhile parsing property \"objectSet\" of s
      tatic type ArrayOfObjectSpec\n\nwhile parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterSpe
      c\nat line 1, column 205\n\nwhile parsing call information for method RetrieveProperties\nat line 1, column 110\n\n
      while parsing SOAP body\nat line 1, column 102\n\nwhile parsing SOAP envelope\nat line 1, column 38\n\nwhile parsin
      g HTTP request for method retrieveContents\non object of type vmodl.query.PropertyCollector\nat line 1, column 0\n"
      ,"wait":0}}] }

      2013-09-21 06:56:18,264 DEBUG [agent.transport.Request] (Job-Executor-36:job-88 = [ c7cbee54-930d-4020-ae02-4710806
      bdc00 ]) Seq 9-1758069619: Received: { Ans: , MgmtId: 161197867246747, via: 9, Ver: v1, Flags: 110,

      { CopyCmdAnswe r }

      }
      2013-09-21 06:56:18,270 DEBUG [agent.manager.AgentAttache] (AgentManager-Handler-14:null) Seq 9-1758069619: No more
      commands found
      2013-09-21 06:56:18,291 DEBUG [storage.snapshot.SnapshotManagerImpl] (Job-Executor-36:job-88 = [ c7cbee54-930d-4020
      -ae02-4710806bdc00 ]) Failed to create snapshot
      com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception: Exception: javax.xml.ws.soap.SOAPFaultE
      xception
      Message:
      Required property obj is missing from data object of type ObjectSpec

      while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec
      at line 1, column 305

      while parsing property "objectSet" of static type ArrayOfObjectSpec

      while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterSpec
      at line 1, column 205

      while parsing call information for method RetrieveProperties
      at line 1, column 110

      while parsing SOAP body
      at line 1, column 102

      while parsing SOAP envelope
      at line 1, column 38

      while parsing HTTP request for method retrieveContents
      on object of type vmodl.query.PropertyCollector
      at line 1, column 0

      at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrateg
      y.java:139)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.
      java:277)
      at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
      at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentI
      nstantiationPostProcessor.java:125)
      at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1302)
      at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2739)
      at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
      at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)
      2013-09-21 06:56:18,299 DEBUG [storage.volume.VolumeServiceImpl] (Job-Executor-36:job-88 = [ c7cbee54-930d-4020-ae0
      2-4710806bdc00 ]) Take snapshot: 14 failed
      com.cloud.utils.exception.CloudRuntimeException: Failed to create snapshot
      at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1040)
      at com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentI
      nstantiationPostProcessor.java:125)
      at org.apache.cloudstack.storage.volume.VolumeServiceImpl.takeSnapshot(VolumeServiceImpl.java:1302)
      at com.cloud.storage.VolumeManagerImpl.takeSnapshot(VolumeManagerImpl.java:2739)
      at org.apache.cloudstack.api.command.user.snapshot.CreateSnapshotCmd.execute(CreateSnapshotCmd.java:170)
      at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:158)
      at com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:531)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
      at java.util.concurrent.FutureTask.run(FutureTask.java:166)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
      at java.lang.Thread.run(Thread.java:679)
      Caused by: com.cloud.utils.exception.CloudRuntimeException: backup snapshot exception: Exception: javax.xml.ws.soap
      .SOAPFaultException
      Message:

      Required property obj is missing from data object of type ObjectSpec

      while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec
      at line 1, column 305

      while parsing property "objectSet" of static type ArrayOfObjectSpec

      while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterSpec
      at line 1, column 205

      while parsing call information for method RetrieveProperties
      at line 1, column 110

      while parsing SOAP body
      at line 1, column 102

      while parsing SOAP envelope
      at line 1, column 38

      while parsing HTTP request for method retrieveContents
      on object of type vmodl.query.PropertyCollector
      at line 1, column 0

      at org.apache.cloudstack.storage.snapshot.SnapshotServiceImpl.backupSnapshot(SnapshotServiceImpl.java:280)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.backupSnapshot(XenserverSnapshotStrategy.java:139)
      at org.apache.cloudstack.storage.snapshot.XenserverSnapshotStrategy.takeSnapshot(XenserverSnapshotStrategy.java:277)
      at com.cloud.storage.snapshot.SnapshotManagerImpl.takeSnapshot(SnapshotManagerImpl.java:1013)
      ... 16 more
      2013-09-21 06:56:18,300 DEBUG [cloud.async.AsyncJobManagerImpl] (Job-Executor-36:job-88 = [ c7cbee54-930d-4020-ae02-4710806bdc00 ]) Complete async job-88 = [ c7cbee54-930d-4020-ae02-4710806bdc00 ], jobStatus: 2, resultCode: 530, result: Error Code: 530 Error text: Failed to create snapshot due to an internal error creating snapshot for volume 14
      2013-09-21 06:56:20,404 DEBUG [cloud.api.ApiServlet] (catalina-exec-14:null) ===START=== 10.216.51.147 – GET command=queryAsyncJobResult&jobId=c7cbee54-930d-4020-ae02-4710806bdc00&response=json&sessionkey=lJPNS2cs%2FvyeTKM2Si1gP8tBG1w%3D&_=1379772417612
      2013-09-21 06:56:20,413 DEBUG [cloud.async.AsyncJobManagerImpl] (catalina-exec-14:null) Async job-88 = [ c7cbee54-930d-4020-ae02-4710806bdc00 ] completed

      In the logs I see the following exceptions from ~9:15 yesterday for GetStorageStatsCommand .
      Also see same exceptions for “GetHostStatsCommand”

      2013-09-20 21:26:23,252 ERROR [vmware.resource.VmwareResource] (DirectAgent-302:10.223.57.194) Unable to execute Ge
      tStorageStatsCommand(storageId : 39e98257-9a36-3a72-afdc-f128c574c04f, localPath: /export/home/sangeetha/campofinal
      ga/primary2, poolType: NetworkFilesystem) due to Exception: javax.xml.ws.soap.SOAPFaultException
      Message:
      Required property obj is missing from data object of type ObjectSpec

      while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec
      at line 1, column 305

      while parsing property "objectSet" of static type ArrayOfObjectSpec

      while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterSpec
      at line 1, column 205

      while parsing call information for method RetrieveProperties
      at line 1, column 110

      while parsing SOAP body
      at line 1, column 102

      while parsing SOAP envelope
      at line 1, column 38

      while parsing HTTP request for method retrieveContents
      on object of type vmodl.query.PropertyCollector
      at line 1, column 0

      javax.xml.ws.soap.SOAPFaultException:
      Required property obj is missing from data object of type ObjectSpec

      while parsing serialized DataObject of type vmodl.query.PropertyCollector.ObjectSpec
      at line 1, column 305

      while parsing property "objectSet" of static type ArrayOfObjectSpec

      while parsing serialized DataObject of type vmodl.query.PropertyCollector.FilterSpec

      Attachments

        1. temp.rar
          5.89 MB
          Sangeetha Hariharan

        Activity

          People

            kelveny Kelven Yang
            sangeethah Sangeetha Hariharan
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: