Uploaded image for project: 'CloudStack'
  1. CloudStack
  2. CLOUDSTACK-1967

Maintenance mode not fully working with VMWare

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 4.1.0, 4.2.0
    • Future
    • VMware
    • Security Level: Public (Anyone can view this level - this is the default.)
    • None

    Description

      When we turn on maintenance mode on one node of the cluster, the VMs are migrating fine, but the host is never fully sent to maintenance mode on the VMWare side. We do see bunch of stack traces related to the management server loosing connectivity, but machine will be available to ICMP for the entire process. The host will appear as Disconnected. This is not a critical issue (major tho) since the VMs are migrated properly, but the error should be investigated/reproduced.

      See the attachments for logs.

      Catalina.out StackTrace:
      INFO [vmware.resource.VmwareResource] (ClusteredAgentManager Timer VmwareResource network configuration info. private vSwitch: vSwitch0, public vSwitch: vSwitch0, guest network: vSwitch0
      INFO [vmware.resource.VmwareResource] (AgentTaskPool-6 Startup report on host hardware info.

      {"memoryBytes":68708646912,"cpuCount":8,"cpuSpeed":2800}

      INFO [vmware.resource.VmwareResource] (AgentTaskPool-6 Startup report on host network info.

      {"hostIp":"10.0.22.65","hostNetmask":"255.255.255.0","hostMacAddress":"00:26:55:2b:c0:8a"}

      INFO [vmware.resource.VmwareResource] (AgentTaskPool-6 Add local storage startup command: {"totalSize":0,"poolInfo":

      {"uuid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9206","poolType":"LVM","capacityBytes":67914170368,"availableBytes":66896003072}

      ,"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","name":"4e656219-c3f9-4a7d-93e3-5c4d09e8c59c","wait":0}
      INFO [vmware.resource.VmwareResource] (AgentTaskPool-6 Add local storage startup command: {"totalSize":0,"poolInfo":

      {"uuid":"ddd756cb-8592-4473-9482-5a4b68085a05","host":"yul01vi13.ops.dot","localPath":"","hostPath":"datastore-9221","poolType":"LVM","capacityBytes":733634101248,"availableBytes":732612788224}

      ,"resourceType":"STORAGE_POOL","hostDetails":{},"type":"Storage","dataCenter":"3","pod":"3","cluster":"12","guid":"ddd756cb-8592-4473-9482-5a4b68085a05","name":"ddd756cb-8592-4473-9482-5a4b68085a05","wait":0}
      INFO [agent.manager.DirectAgentAttache] (AgentTaskPool-6 StartupAnswer received 4 Interval = 60
      INFO [vmware.resource.VmwareResource] (DirectAgent-494:yul01vi13.ops.dot) Executing resource CheckNetworkCommand {"networkInfoList":[

      {"physicalNetworkId":206,"privateNetworkName":"vSwitch0"}

      ,

      {"physicalNetworkId":207,"storageNetworkName":"vSwitch1"}

      ,

      {"physicalNetworkId":208,"publicNetworkName":"vSwitch2","guestNetworkName":"vSwitch2"}

      ],"wait":0}
      INFO [network.security.SecurityGroupListener] (AgentTaskPool-6 Received a host startup notification
      INFO [network.security.SecurityGroupListener] (AgentTaskPool-6 Scheduled network rules cleanup, interval=1961
      INFO [network.security.SecurityGroupListener] (AgentTaskPool-6 Received a host startup notification
      INFO [network.security.SecurityGroupListener] (AgentTaskPool-6 Received a host startup notification
      WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6 Resource [Host:4] is unreachable: Host 4: Unable to send class com.cloud.agent.api.ModifyStoragePoolCommand because agent is in maintenance mode
      WARN [storage.listener.StoragePoolMonitor] (AgentTaskPool-6 Unable to connect host 4 to pool Pool[205|VMFS] due to com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205] is unreachable: Unable to get an answer to the modify storage pool command
      com.cloud.exception.StorageUnavailableException: Resource [StoragePool:205] is unreachable: Unable to get an answer to the modify storage pool command
      at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
      at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
      at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
      at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
      at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
      at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
      at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
      at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
      at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
      at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
      at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
      at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
      at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:679)
      WARN [agent.manager.AgentManagerImpl] (AgentTaskPool-6 Resource [Host:4] is unreachable: Host 4: Unable to send class com.cloud.agent.api.ModifyStoragePoolCommand because agent is in maintenance mode
      WARN [storage.listener.StoragePoolMonitor] (AgentTaskPool-6 Unable to connect host 4 to pool Pool[206|VMFS] due to com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206] is unreachable: Unable to get an answer to the modify storage pool command
      com.cloud.exception.StorageUnavailableException: Resource [StoragePool:206] is unreachable: Unable to get an answer to the modify storage pool command
      at com.cloud.storage.StorageManagerImpl.connectHostToSharedPool(StorageManagerImpl.java:1615)
      at com.cloud.storage.listener.StoragePoolMonitor.processConnect(StoragePoolMonitor.java:94)
      at com.cloud.agent.manager.AgentManagerImpl.notifyMonitorsOfConnection(AgentManagerImpl.java:611)
      at com.cloud.agent.manager.AgentManagerImpl.handleDirectConnectAgent(AgentManagerImpl.java:1504)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
      at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
      at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
      at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      at sun.proxy.$Proxy109.handleDirectConnectAgent(Unknown Source)
      at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:1937)
      at com.cloud.resource.ResourceManagerImpl.createHostAndAgent(ResourceManagerImpl.java:2074)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:319)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
      at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
      at com.cloud.utils.db.TransactionContextBuilder.AroundAnyMethod(TransactionContextBuilder.java:45)
      at sun.reflect.GeneratedMethodAccessor27.invoke(Unknown Source)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
      at java.lang.reflect.Method.invoke(Method.java:616)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
      at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
      at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:90)
      at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
      at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:202)
      at sun.proxy.$Proxy119.createHostAndAgent(Unknown Source)
      at com.cloud.agent.manager.AgentManagerImpl$SimulateStartTask.run(AgentManagerImpl.java:1156)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:679)
      INFO [vmware.resource.VmwareResource] (DirectAgent-403:yul01vi13.ops.dot) Executing resource ReadyCommand:

      {"dcId":3,"hostId":4,"wait":0}

      INFO [agent.manager.AgentMonitor] (Thread-5 Found the following agents behind on ping: [8, 4]
      INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-7 Host 4 is disconnecting with event ShutdownRequested
      INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10 Investigating why host 8 has disconnected with event PingTimeout
      INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10 The state determined is Up
      INFO [agent.manager.AgentManagerImpl] (AgentTaskPool-10 Agent is determined to be up and running

      Attachments

        1. managementlog_maint.log
          40 kB
          Francois Gaudreault

        Activity

          People

            sateeshc Sateesh Chodapuneedi
            fgaudreault Francois Gaudreault
            Votes:
            1 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: