Details
-
Bug
-
Status: Resolved
-
Blocker
-
Resolution: Cannot Reproduce
-
4.1.0 Beta
-
None
Description
Hi,
We are testing the behavior of stratos 4.1.0 rest api's and found that if we issue an undeploy-application followed immediately with application-remove, that we see the following traceback.
The VM successfully get's deleted but not sure what kind of side effects this has on the system.
Here's the snippet from the wso2carbon.log file. I've also attached the entire log.
Steps to reproduce this:
1) deploy an application which leads to startup a single instance of a cartridge.
2) wait for it to become active
3) issue the undeploy application and application remove rest api calls.
-Vanson
TID: [0] [STRATOS] [2015-03-19 19:45:08,681] INFO
{org.wso2.carbon.databridge.core.DataBridge} - admin connected
TID: [0] [STRATOS] [2015-03-19 19:45:43,023] INFO
TID: [0] [STRATOS] [2015-03-19 19:45:43,024] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl}
- Removing application signup: [application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19 19:45:43,051] INFO
signup: [application-id] cisco-sample-vm [tenant-id] -1234
TID: [0] [STRATOS] [2015-03-19 19:45:43,084] INFO {org.apache.stratos.manager.components.ApplicationSignUpHandler}
- Application signup removed successfully: [application-id] cisco-sample-vm [tenant-id] -1234
TID: [0] [STRATOS] [2015-03-19 19:45:43,091] INFO
- Network partition algorithm context is removed successfully: [id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19 19:45:43,092] INFO
- Publishing Cluster terminating event for [application] cisco-sample-vm [cluster] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19 19:45:43,109] INFO
TID: [0] [STRATOS] [2015-03-19 19:45:43,117] INFO {org.apache.stratos.autoscaler.applications.topic.ApplicationsEventPublisher} - Publishing application inactivated event: [application] cisco-sample-vm [instance] cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19 19:45:43,120] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing Cluster terminating event: [application-id] cisco-sample-vm [cluster id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
[instance-id] cisco-sample-vm-1
TID: [0] [STRATOS] [2015-03-19 19:45:43,144] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application undeployed successfully: [application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19 19:45:43,154] INFO {org.apache.stratos.autoscaler.event.receiver.topology.AutoscalerTopologyEventReceiver}
- [ClusterTerminatingEvent] Received: class org.apache.stratos.messaging.event.topology.ClusterInstanceTerminatingEvent
TID: [0] [STRATOS] [2015-03-19 19:45:43,155] INFO {org.apache.stratos.autoscaler.event.publisher.InstanceNotificationPublisher} - Publishing Instance Cleanup Event: [cluster] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain
TID: [0] [STRATOS] [2015-03-19 19:45:43,171] WARN {org.apache.stratos.autoscaler.status.processor.cluster.ClusterStatusActiveProcessor} - No possible state change found for [type] [cluster] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [instance]
TID: [0] [STRATOS] [2015-03-19 19:45:43,192] INFO {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder}
- member maintenance mode event adding status started
TID: [0] [STRATOS] [2015-03-19 19:45:43,204] INFO
TID: [0] [STRATOS] [2015-03-19 19:45:43,230] INFO {org.apache.stratos.messaging.message.processor.topology.MemberMaintenanceModeProcessor}
- Member updated as In_Maintenance: [service] cisco-sample-vm [cluster] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:43,231] INFO {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - Member Ready to shut down event adding status started
TID: [0] [STRATOS] [2015-03-19 19:45:43,242] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher}
- Publishing member ready to shut down event: [service-name] cisco-sample-vm [cluster-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [cluster-instance-id] cisco-sample-vm-1 [member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d [network-partition-id] RegionOne [partition-id] whole-region
TID: [0] [STRATOS] [2015-03-19 19:45:43,262] INFO
- Member updated as Ready to shutdown:
[service] cisco-sample-vm [cluster] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain [member] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO
- [MemberReadyToShutdownEvent] Received: class org.apache.stratos.messaging.event.topology.MemberReadyToShutdownEvent
TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext} - Starting the moving of termination pending to obsolete for [member] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:43,264] INFO {org.apache.stratos.autoscaler.context.partition.ClusterLevelPartitionContext}- Found termination pending member and trying to move [member] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d to obsolete list
{org.apache.stratos.common.client.CloudControllerServiceClient} - Terminating instance via cloud controller: [member] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:43,266] INFO
TID: [0] [STRATOS] [2015-03-19 19:45:43,275] INFO {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Starting to terminate
member: [cartridge-type] cisco-sample-vm [member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:43,883] INFO {org.apache.stratos.messaging.domain.application.locking.ApplicationLockHierarchy} - Removed lock for application: [application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO {org.apache.stratos.cloud.controller.messaging.receiver.application.ApplicationEventReceiver} - Application deleted event received:
[application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19 19:45:43,909] INFO {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - Removed application [ cisco-sample-vm ]'s Cluster [ cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain ] from the topology
TID: [0] [STRATOS] [2015-03-19 19:45:43,910] INFO {org.apache.stratos.autoscaler.services.impl.AutoscalerServiceImpl} - Application deleted successfully: [application-id]
TID: [0] [STRATOS] [2015-03-19 19:45:43,930] INFO {org.apache.stratos.cloud.controller.messaging.publisher.TopologyEventPublisher} - Publishing application clusters removed event: [application-id] cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19 19:45:43,950] INFO {org.apache.stratos.messaging.message.processor.topology.ApplicationClustersRemovedMessageProcessor} - Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain removed from topology for application cisco-sample-vm
TID: [0] [STRATOS] [2015-03-19 19:45:43,952] INFO {org.apache.stratos.messaging.message.processor.application.ApplicationDeletedMessageProcessor} - [Application] cisco-sample-vm has been successfully removed
TID: [0] [STRATOS] [2015-03-19 19:45:44,786] INFO {org.apache.stratos.cloud.controller.iaases.JcloudsIaas} - Member terminated:
[member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:44,786] WARN {org.apache.stratos.cloud.controller.messaging.topology.TopologyBuilder} - Cluster cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain does not exist
TID: [0] [STRATOS] [2015-03-19 19:45:50,012] INFO {org.apache.stratos.common.client.CloudControllerServiceClient}- Terminating instance via cloud controller: [member] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
{org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl} - Could not terminate instance, member context not found: [member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
TID: [0] [STRATOS] [2015-03-19 19:45:50,017] ERROR
TID: [0] [STRATOS] [2015-03-19 19:45:50,018] ERROR {org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl}- Could not terminate instance: [member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
{org.apache.stratos.common.concurrent.locks.ReadWriteLock}
org.apache.stratos.cloud.controller.exception.InvalidMemberException: Could not terminate instance, member context not found: [member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
at org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
TID: [0] [STRATOS] [2015-03-19 19:45:50,019] WARN- System warning! Trying to release a lock which has not been taken by the same thread: [lock-name] topology-manager [thread-id] 164 [thread-name] http-nio-9443-exec-29
{org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver}
TID: [0] [STRATOS] [2015-03-19 19:45:50,020] ERROR- Could not terminate instance, member context not found: [member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
{org.apache.stratos.autoscaler.rule.RuleTasksDelegator}
org.apache.stratos.cloud.controller.exception.InvalidMemberException: Could not terminate instance, member context not found: [member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
at org.apache.stratos.cloud.controller.services.impl.CloudControllerServiceImpl.terminateInstance(CloudControllerServiceImpl.java:553)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.axis2.rpc.receivers.RPCUtil.invokeServiceClass(RPCUtil.java:212)
at org.apache.axis2.rpc.receivers.RPCInOnlyMessageReceiver.invokeBusinessLogic(RPCInOnlyMessageReceiver.java:66)
at org.apache.axis2.receivers.AbstractMessageReceiver.receive(AbstractMessageReceiver.java:110)
at org.apache.axis2.engine.AxisEngine.receive(AxisEngine.java:180)
at org.apache.axis2.transport.http.HTTPTransportUtils.processHTTPPostRequest(HTTPTransportUtils.java:172)
at org.apache.axis2.transport.http.AxisServlet.doPost(AxisServlet.java:146)
at org.wso2.carbon.core.transports.CarbonServlet.doPost(CarbonServlet.java:231)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:755)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.eclipse.equinox.http.servlet.internal.ServletRegistration.service(ServletRegistration.java:61)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.processAlias(ProxyServlet.java:128)
at org.eclipse.equinox.http.servlet.internal.ProxyServlet.service(ProxyServlet.java:68)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
at org.wso2.carbon.tomcat.ext.servlet.DelegationServlet.service(DelegationServlet.java:68)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.wso2.carbon.tomcat.ext.filter.CharacterSetFilter.doFilter(CharacterSetFilter.java:61)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:222)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:123)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:171)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:99)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.continueInvocation(CompositeValve.java:178)
at org.wso2.carbon.tomcat.ext.valves.CarbonTomcatValve$1.invoke(CarbonTomcatValve.java:47)
at org.wso2.carbon.webapp.mgt.TenantLazyLoaderValve.invoke(TenantLazyLoaderValve.java:56)
at org.wso2.carbon.tomcat.ext.valves.TomcatValveContainer.invokeValves(TomcatValveContainer.java:47)
at org.wso2.carbon.tomcat.ext.valves.CompositeValve.invoke(CompositeValve.java:141)
at org.wso2.carbon.tomcat.ext.valves.CarbonStuckThreadDetectionValve.invoke(CarbonStuckThreadDetectionValve.java:156)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:936)
at org.wso2.carbon.tomcat.ext.valves.CarbonContextCreatorValve.invoke(CarbonContextCreatorValve.java:52)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1004)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:589)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.run(NioEndpoint.java:1653)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
TID: [0] [STRATOS] [2015-03-19 19:45:50,027] ERROR- Cannot terminate instance
org.apache.axis2.AxisFault: Could not terminate instance, member context not found: [member-id] cisco-sample-vm.cisco-sample-vm.cisco-sample-vm.domain41624077-30c4-4bfe-a2a8-ece2fc4f550d
at org.apache.axis2.util.Utils.getInboundFaultFromMessageContext(Utils.java:531)
at
org.apache.axis2.description.RobustOutOnlyAxisOperation$RobustOutOnlyOperationClient.handleResponse(RobustOutOnlyAxisOperation.java:91)
at org.apache.axis2.description.OutInAxisOperationClient.send(OutInAxisOperation.java:445)
at org.apache.axis2.description.OutInAxisOperationClient.executeImpl(OutInAxisOperation.java:225)
at org.apache.axis2.client.OperationClient.execute(OperationClient.java:149)
at org.apache.stratos.cloud.controller.stub.CloudControllerServiceStub.terminateInstance(CloudControllerServiceStub.java:7071)
at org.apache.stratos.common.client.CloudControllerServiceClient.terminateInstance(CloudControllerServiceClient.java:120)
at org.apache.stratos.autoscaler.rule.RuleTasksDelegator.terminateObsoleteInstance(RuleTasksDelegator.java:311)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.mvel2.optimizers.impl.refl.nodes.MethodAccessor.getValue(MethodAccessor.java:48)
at org.mvel2.optimizers.impl.refl.nodes.VariableAccessor.getValue(VariableAccessor.java:37)
at org.mvel2.ast.ASTNode.getReducedValueAccelerated(ASTNode.java:108)
at org.mvel2.MVELRuntime.execute(MVELRuntime.java:85)
at org.mvel2.compiler.CompiledExpression.getDirectValue(CompiledExpression.java:123)
at org.mvel2.compiler.CompiledExpression.getValue(CompiledExpression.java:119)
at org.mvel2.MVEL.executeExpression(MVEL.java:930)
at org.drools.base.mvel.MVELConsequence.evaluate(MVELConsequence.java:104)
at org.drools.common.DefaultAgenda.fireActivation(DefaultAgenda.java:1287)
at org.drools.common.DefaultAgenda.fireNextItem(DefaultAgenda.java:1221)