[CLOUDSTACK-9726] state of the rvr dose not change to update failed when updating rvrs in sequence to a new offering fails.

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 4.9.0
• Fix Version/s: 4.9.0
• Component/s: Management Server
• Security Level: Public (Anyone can view this level - this is the default.)
• Assignee: Bharat Kumar
• Reporter: Bharat Kumar
• Created: 02/Jan/17 10:43
• Updated: 17/May/17 05:53

Description

state of the rvr dose not change to update failed when updating rvrs in sequence to a new offering fails.

Create two Network offerings - RVR1 , RVR2 with RVR enabled.
Deploy an instance by selecting RVR1 offering for the new Network and ensure virtual routers are deployed and are in Running State.
Update the network by setting the parameter Updateinsequence to true with offering ID RVR2.
While the Network update is in progress , put the Primary Storage in Maintenance Mode.

Observations:

Pre-Network Update states :

VR 1 - UPdate Complete , VR 2 - Update in Progress

States after Host in maintenance mode :

Routers moved to stopped state.

VR 1 - Update needed , VR 2 , VR3 : Update in Progress .

States after cancelling maintenance mode:

VR 1 - Update needed , VR 2 , VR 3 : Update in Progress .

Events shows Network update to new offering failed and following exceptions are seen in logs .

Execpetion during host in maintenance mode :

2016-01-01 08:44:44,453 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-21:ctx-96b2541b job-49/job-56) (logid:b50e7b75) Remove job-56 from job monitoring
2016-01-01 08:44:44,458 DEBUG [c.c.n.NetworkModelImpl] (API-Job-Executor-23:ctx-febc4ccc job-49 ctx-82bfdcbd) (logid:b50e7b75) Service SecurityGroup is not supported in the network id=205
2016-01-01 08:44:44,470 WARN [c.c.n.NetworkServiceImpl] (API-Job-Executor-23:ctx-febc4ccc job-49 ctx-82bfdcbd) (logid:b50e7b75) Failed to implement network Ntwk[205|Guest|17] elements and resources as a part of network update due to
com.cloud.exception.AgentUnavailableException: Resource [Host:1] is unreachable: Host 1: Unable to send class com.cloud.agent.api.routing.AggregationControlCommand because agent perf04 is in maintenance mode
at com.cloud.agent.manager.AgentAttache.checkAvailability(AgentAttache.java:165)
at com.cloud.agent.manager.ClusteredAgentAttache.checkAvailability(ClusteredAgentAttache.java:82)
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:346)
at com.cloud.agent.manager.ClusteredAgentAttache.send(ClusteredAgentAttache.java:141)
at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:395)
at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:457)
at com.cloud.agent.manager.AgentManagerImpl.send(AgentManagerImpl.java:974)
at com.cloud.network.router.NetworkHelperImpl.sendCommandsToRouter(NetworkHelperImpl.java:180)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.aggregationExecution(VirtualNetworkApplianceManagerImpl.java:2765)
at com.cloud.network.router.VirtualNetworkApplianceManagerImpl.prepareAggregatedExecution(VirtualNetworkApplianceManagerImpl.java:2778)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy218.prepareAggregatedExecution(Unknown Source)
at com.cloud.network.element.VirtualRouterElement.prepareAggregatedExecution(VirtualRouterElement.java:1271)
at org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.implementNetworkElementsAndResources(NetworkOrchestrator.java:1135)
at com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2379)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy162.updateGuestNetwork(Unknown Source)
at org.apache.cloudstack.api.command.admin.network.UpdateNetworkCmdByAdmin.execute(UpdateNetworkCmdByAdmin.java:51)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:558)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:505)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2016-01-01 08:44:44,478 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-23:ctx-febc4ccc job-49) (logid:b50e7b75) Unexpected exception while executing org.apache.cloudstack.api.command.admin.network.UpdateNetworkCmdByAdmin
com.cloud.utils.exception.CloudRuntimeException: failed to update network 99123d1d-53f7-4833-9b42-4972dca1303fdue to Failed to implement network (with specified id) elements and resources as a part of network update
at com.cloud.network.NetworkServiceImpl.updateGuestNetwork(NetworkServiceImpl.java:2410)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:601)
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317)
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150)
at org.apache.cloudstack.network.contrail.management.EventUtils$EventInterceptor.invoke(EventUtils.java:106)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInterceptor.java:51)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204)
at $Proxy162.updateGuestNetwork(Unknown Source)
at org.apache.cloudstack.api.command.admin.network.UpdateNetworkCmdByAdmin.execute(UpdateNetworkCmdByAdmin.java:51)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:108)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:558)
at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:505)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
2016-01-01 08:44:44,480 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:ctx-febc4ccc job-49) (logid:b50e7b75) Complete async job-49, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/

{"uuidList":[],"errorcode":530,"errortext":"failed to update network 99123d1d-53f7-4833-9b42-4972dca1303fdue to Failed to implement network (with specified id) elements and resources as a part of network update"}

2016-01-01 08:44:44,481 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-23:ctx-febc4ccc job-49) (logid:b50e7b75) Publish async job-49 complete on message bus

Issue Links

• links to: GitHub Pull Request #1971