ActiveMQ
  1. ActiveMQ
  2. AMQ-1862

Master broker is down when slave broker is crashed

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Won't Fix
    • Affects Version/s: 5.0.0, 5.1.0
    • Fix Version/s: NEEDS_REVIEW
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      master broker(AMQ5.0) running on the Winxp+sp2 (JDK1.6)
      slave broker(AMQ5.0) running on the windows2003(JDK1.6)

      Description

      I have met the bugs of pure master/slave clustering on the AMQ 5.0.0. The issue is that master broker is down when I kill the slave broker ungracefully. I can't access AMQ web page(http://localhost:8161/admin/queues.jsp) and can't send any messages to MQ using "ant producer" provided by AMQ examples.
      I think that this case is very common in the production env. Thank your very much.

      Test scenarios:
      1. Start master broker and slave broker orderly
      2. Kill slave broker to simulate machine crash suddenly
      3. Send a lot of messages to MQ
      ant producer

      Test result:
      master MQ throws Failed to add Connection exception and master MQ broker is down since I can not access the queue monitor page of master MQ web console.

      The error message represented on the queue page(http://localhost:8161/admin/queues.jsp):
      ==============================================================
      HTTP ERROR: 500

      java.lang.reflect.UndeclaredThrowableException

      RequestURI=/admin/queues.jsp
      Caused by:

      javax.el.ELException: java.lang.reflect.UndeclaredThrowableException
      at javax.el.BeanELResolver.getValue(BeanELResolver.java:266)
      at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:143)
      at com.sun.el.parser.AstValue.getValue(AstValue.java:125)
      at com.sun.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:195)
      at org.apache.jasper.runtime.PageContextImpl.evaluateExpression(PageContextImpl.java:1008)
      at org.apache.jsp.queues_jsp._jspx_meth_c_forEach_0(org.apache.jsp.queues_jsp:139)
      at org.apache.jsp.queues_jsp._jspService(org.apache.jsp.queues_jsp:99)
      at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:80)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
      at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:464)
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:358)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1098)
      at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:63)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.apache.activemq.web.SessionFilter.doFilter(SessionFilter.java:46)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.apache.activemq.web.filter.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:81)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:118)
      at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
      at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
      at org.mortbay.jetty.Server.handle(Server.java:295)
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:503)
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:827)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:511)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:210)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:379)
      at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:361)
      at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
      Caused by: java.lang.reflect.UndeclaredThrowableException
      at $Proxy21.getName(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at javax.el.BeanELResolver.getValue(BeanELResolver.java:261)
      ... 39 more
      Caused by: javax.management.InstanceNotFoundException: org.apache.activemq:BrokerName=master,Type=Queue,Destination=example.A
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
      at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:263)
      ... 45 more

      Caused by:

      java.lang.reflect.UndeclaredThrowableException
      at $Proxy21.getName(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at javax.el.BeanELResolver.getValue(BeanELResolver.java:261)
      at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:143)
      at com.sun.el.parser.AstValue.getValue(AstValue.java:125)
      at com.sun.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:195)
      at org.apache.jasper.runtime.PageContextImpl.evaluateExpression(PageContextImpl.java:1008)
      at org.apache.jsp.queues_jsp._jspx_meth_c_forEach_0(org.apache.jsp.queues_jsp:139)
      at org.apache.jsp.queues_jsp._jspService(org.apache.jsp.queues_jsp:99)
      at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:80)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
      at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:464)
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:358)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1098)
      at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:63)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.apache.activemq.web.SessionFilter.doFilter(SessionFilter.java:46)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.apache.activemq.web.filter.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:81)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:118)
      at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
      at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
      at org.mortbay.jetty.Server.handle(Server.java:295)
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:503)
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:827)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:511)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:210)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:379)
      at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:361)
      at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)
      Caused by: javax.management.InstanceNotFoundException: org.apache.activemq:BrokerName=master,Type=Queue,Destination=example.A
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
      at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:263)
      ... 45 more

      Caused by:

      javax.management.InstanceNotFoundException: org.apache.activemq:BrokerName=master,Type=Queue,Destination=example.A
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getMBean(DefaultMBeanServerInterceptor.java:1094)
      at com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:662)
      at com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:638)
      at javax.management.MBeanServerInvocationHandler.invoke(MBeanServerInvocationHandler.java:263)
      at $Proxy21.getName(Unknown Source)
      at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
      at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
      at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
      at java.lang.reflect.Method.invoke(Method.java:597)
      at javax.el.BeanELResolver.getValue(BeanELResolver.java:261)
      at javax.el.CompositeELResolver.getValue(CompositeELResolver.java:143)
      at com.sun.el.parser.AstValue.getValue(AstValue.java:125)
      at com.sun.el.ValueExpressionImpl.getValue(ValueExpressionImpl.java:195)
      at org.apache.jasper.runtime.PageContextImpl.evaluateExpression(PageContextImpl.java:1008)
      at org.apache.jsp.queues_jsp._jspx_meth_c_forEach_0(org.apache.jsp.queues_jsp:139)
      at org.apache.jsp.queues_jsp._jspService(org.apache.jsp.queues_jsp:99)
      at org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:80)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:373)
      at org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:464)
      at org.apache.jasper.servlet.JspServlet.service(JspServlet.java:358)
      at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
      at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:487)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1098)
      at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:63)
      at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.apache.activemq.web.SessionFilter.doFilter(SessionFilter.java:46)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.apache.activemq.web.filter.ApplicationContextFilter.doFilter(ApplicationContextFilter.java:81)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at com.opensymphony.module.sitemesh.filter.PageFilter.parsePage(PageFilter.java:118)
      at com.opensymphony.module.sitemesh.filter.PageFilter.doFilter(PageFilter.java:52)
      at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1089)
      at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:365)
      at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
      at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:181)
      at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:712)
      at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:405)
      at org.mortbay.jetty.handler.HandlerCollection.handle(HandlerCollection.java:114)
      at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:139)
      at org.mortbay.jetty.Server.handle(Server.java:295)
      at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:503)
      at org.mortbay.jetty.HttpConnection$RequestHandler.headerComplete(HttpConnection.java:827)
      at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:511)
      at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:210)
      at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:379)
      at org.mortbay.io.nio.SelectChannelEndPoint.run(SelectChannelEndPoint.java:361)
      at org.mortbay.thread.BoundedThreadPool$PoolThread.run(BoundedThreadPool.java:442)

      Powered by Jetty://
      ==============================================================

      The following error is the logs on the master broker:
      ==============================================================
      2008-07-17 10:51:27,965 [main ] INFO BrokerService - Using Persistence Adapter: AMQPersistenceAdapter(D:\apache-activemq-5.0.0\bin\..\data\master)
      2008-07-17 10:51:27,981 [main ] INFO BrokerService - ActiveMQ 5.0.0 JMS Message Broker (master) is starting
      2008-07-17 10:51:27,981 [main ] INFO BrokerService - For help or more information please see: http://activemq.apache.org/
      2008-07-17 10:51:28,028 [main ] INFO AMQPersistenceAdapter - AMQStore starting using directory: D:\apache-activemq-5.0.0\bin\..\data\master
      2008-07-17 10:51:28,043 [JMX connector ] INFO ManagementContext - JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi
      2008-07-17 10:51:28,059 [main ] INFO KahaStore - Kaha Store using data directory D:\apache-activemq-5.0.0\bin\..\data\master\kr-store\state
      2008-07-17 10:51:28,106 [main ] INFO AMQPersistenceAdapter - Active data files: []
      2008-07-17 10:51:28,324 [main ] INFO KahaStore - Kaha Store using data directory D:\apache-activemq-5.0.0\bin\..\data\master\kr-store\data
      2008-07-17 10:51:28,340 [main ] INFO TransportServerThreadSupport - Listening for connections at: tcp://QLIN:61616
      2008-07-17 10:51:28,340 [main ] INFO TransportConnector - Connector openwire Started
      2008-07-17 10:51:28,340 [main ] INFO BrokerService - ActiveMQ JMS Message Broker (master, ID:QLIN-41543-1216263088199-0:0) started
      2008-07-17 10:51:28,356 [main ] WARN BrokerRegistry - Broker localhost not started so using master instead
      2008-07-17 10:51:28,356 [main ] INFO TransportConnector - Connector vm://localhost Started
      2008-07-17 10:51:28,481 [main ] INFO log - Logging to org.slf4j.impl.JCLLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
      2008-07-17 10:51:28,496 [main ] INFO log - jetty-6.1.4
      2008-07-17 10:51:28,762 [main ] INFO WebConsoleStarter - ActiveMQ WebConsole initialized.
      2008-07-17 10:51:28,840 [main ] INFO /admin - Initializing Spring FrameworkServlet 'dispatcher'
      2008-07-17 10:51:28,903 [main ] INFO log - ActiveMQ Console at http://0.0.0.0:8161/admin
      2008-07-17 10:51:28,996 [main ] INFO log - ActiveMQ Web Demos at http://0.0.0.0:8161/demo
      2008-07-17 10:51:29,012 [main ] INFO log - Started SelectChannelConnector@0.0.0.0:8161
      2008-07-17 10:51:36,543 [10.0.11.35:1483] INFO TransportConnection - Slave Broker slave is attached
      2008-07-17 10:52:23,215 [127.0.0.1:41546] ERROR Service - Async error occurred: javax.jms.JMSException: The destination topic://ActiveMQ.Advisory.Consumer.Queue.example.A does not exist.
      javax.jms.JMSException: The destination topic://ActiveMQ.Advisory.Consumer.Queue.example.A does not exist.
      at org.apache.activemq.broker.region.AbstractRegion.lookup(AbstractRegion.java:371)
      at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:324)
      at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:402)
      at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:293)
      at org.apache.activemq.advisory.AdvisoryBroker.fireConsumerAdvisory(AdvisoryBroker.java:263)
      at org.apache.activemq.advisory.AdvisoryBroker.fireConsumerAdvisory(AdvisoryBroker.java:257)
      at org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
      at org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:113)
      at org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:123)
      at org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:549)
      at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:64)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172)
      at java.lang.Thread.run(Thread.java:619)
      2008-07-17 10:52:23,215 [nerContainer-55] ERROR efaultMessageListenerContainer - Setup of JMS message listener invoker failed - trying to recover
      javax.jms.JMSException: The destination topic://ActiveMQ.Advisory.Queue does not exist.
      at org.apache.activemq.broker.region.AbstractRegion.lookup(AbstractRegion.java:371)
      at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:324)
      at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:402)
      at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:293)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:253)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:248)
      at org.apache.activemq.advisory.AdvisoryBroker.addDestination(AdvisoryBroker.java:149)
      at org.apache.activemq.broker.BrokerFilter.addDestination(BrokerFilter.java:141)
      at org.apache.activemq.broker.MutableBrokerFilter.addDestination(MutableBrokerFilter.java:151)
      at org.apache.activemq.broker.region.AbstractRegion.lookup(AbstractRegion.java:361)
      at org.apache.activemq.broker.region.AbstractRegion.addConsumer(AbstractRegion.java:214)
      at org.apache.activemq.broker.region.RegionBroker.addConsumer(RegionBroker.java:335)
      at org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:85)
      at org.apache.activemq.advisory.AdvisoryBroker.addConsumer(AdvisoryBroker.java:80)
      at org.apache.activemq.broker.BrokerFilter.addConsumer(BrokerFilter.java:85)
      at org.apache.activemq.broker.MutableBrokerFilter.addConsumer(MutableBrokerFilter.java:95)
      at org.apache.activemq.broker.TransportConnection.processAddConsumer(TransportConnection.java:525)
      at org.apache.activemq.command.ConsumerInfo.visit(ConsumerInfo.java:334)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172)
      at java.lang.Thread.run(Thread.java:619)
      2008-07-17 10:52:23,215 [127.0.0.1:61616] WARN ActiveMQConnection - Async exception with no exception listener: javax.jms.JMSException: The destination topic://ActiveMQ.Advisory.Consumer.Queue.example.A does not exist.
      javax.jms.JMSException: The destination topic://ActiveMQ.Advisory.Consumer.Queue.example.A does not exist.
      at org.apache.activemq.broker.region.AbstractRegion.lookup(AbstractRegion.java:371)
      at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:324)
      at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:402)
      at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:293)
      at org.apache.activemq.advisory.AdvisoryBroker.fireConsumerAdvisory(AdvisoryBroker.java:263)
      at org.apache.activemq.advisory.AdvisoryBroker.fireConsumerAdvisory(AdvisoryBroker.java:257)
      at org.apache.activemq.advisory.AdvisoryBroker.removeConsumer(AdvisoryBroker.java:222)
      at org.apache.activemq.broker.BrokerFilter.removeConsumer(BrokerFilter.java:113)
      at org.apache.activemq.broker.MutableBrokerFilter.removeConsumer(MutableBrokerFilter.java:123)
      at org.apache.activemq.broker.TransportConnection.processRemoveConsumer(TransportConnection.java:549)
      at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:64)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172)
      at java.lang.Thread.run(Thread.java:619)
      2008-07-17 10:52:23,231 [127.0.0.1:41546] WARN Service - Failed to remove connection ConnectionInfo

      {commandId = 1, responseRequired = true, connectionId = ID:QLIN-41543-1216263088199-2:1, clientId = ID:QLIN-41543-1216263088199-4:0, userName = null, password = null, brokerPath = null, brokerMasterConnector = false, manageable = true, clientMaster = true}

      javax.jms.JMSException: The destination topic://ActiveMQ.Advisory.Connection does not exist.
      at org.apache.activemq.broker.region.AbstractRegion.lookup(AbstractRegion.java:371)
      at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:324)
      at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:402)
      at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:293)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:253)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:248)
      at org.apache.activemq.advisory.AdvisoryBroker.removeConnection(AdvisoryBroker.java:211)
      at org.apache.activemq.broker.BrokerFilter.removeConnection(BrokerFilter.java:109)
      at org.apache.activemq.broker.MutableBrokerFilter.removeConnection(MutableBrokerFilter.java:119)
      at org.apache.activemq.broker.TransportConnection.processRemoveConnection(TransportConnection.java:697)
      at org.apache.activemq.command.RemoveInfo.visit(RemoveInfo.java:60)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172)
      at java.lang.Thread.run(Thread.java:619)
      2008-07-17 10:52:23,231 [127.0.0.1:41558] WARN TransportConnection - Failed to add Connection
      javax.jms.JMSException: The destination topic://ActiveMQ.Advisory.Connection does not exist.
      at org.apache.activemq.broker.region.AbstractRegion.lookup(AbstractRegion.java:371)
      at org.apache.activemq.broker.region.AbstractRegion.send(AbstractRegion.java:324)
      at org.apache.activemq.broker.region.RegionBroker.send(RegionBroker.java:402)
      at org.apache.activemq.broker.TransactionBroker.send(TransactionBroker.java:224)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:293)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:253)
      at org.apache.activemq.advisory.AdvisoryBroker.fireAdvisory(AdvisoryBroker.java:248)
      at org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:75)
      at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:81)
      at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:91)
      at org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:657)
      at org.apache.activemq.broker.jmx.ManagedTransportConnection.processAddConnection(ManagedTransportConnection.java:86)
      at org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:125)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172)
      at java.lang.Thread.run(Thread.java:619)
      2008-07-17 10:52:23,231 [127.0.0.1:41558] WARN TransportConnection - Failed to add Connection
      javax.jms.InvalidClientIDException: Broker: master - Client: ID:QLIN-41543-1216263088199-4:1 already connected from /127.0.0.1:41558
      at org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:205)
      at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:81)
      at org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:72)
      at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:81)
      at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:91)
      at org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:657)
      at org.apache.activemq.broker.jmx.ManagedTransportConnection.processAddConnection(ManagedTransportConnection.java:86)
      at org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:125)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172)
      at java.lang.Thread.run(Thread.java:619)
      2008-07-17 10:52:28,231 [nerContainer-56] ERROR efaultMessageListenerContainer - Setup of JMS message listener invoker failed - trying to recover
      javax.jms.InvalidClientIDException: Broker: master - Client: ID:QLIN-41543-1216263088199-4:1 already connected from /127.0.0.1:41558
      at org.apache.activemq.broker.region.RegionBroker.addConnection(RegionBroker.java:205)
      at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:81)
      at org.apache.activemq.advisory.AdvisoryBroker.addConnection(AdvisoryBroker.java:72)
      at org.apache.activemq.broker.BrokerFilter.addConnection(BrokerFilter.java:81)
      at org.apache.activemq.broker.MutableBrokerFilter.addConnection(MutableBrokerFilter.java:91)
      at org.apache.activemq.broker.TransportConnection.processAddConnection(TransportConnection.java:657)
      at org.apache.activemq.broker.jmx.ManagedTransportConnection.processAddConnection(ManagedTransportConnection.java:86)
      at org.apache.activemq.command.ConnectionInfo.visit(ConnectionInfo.java:125)
      at org.apache.activemq.broker.TransportConnection.service(TransportConnection.java:281)
      at org.apache.activemq.broker.TransportConnection$1.onCommand(TransportConnection.java:178)
      at org.apache.activemq.transport.TransportFilter.onCommand(TransportFilter.java:67)
      at org.apache.activemq.transport.WireFormatNegotiator.onCommand(WireFormatNegotiator.java:134)
      at org.apache.activemq.transport.InactivityMonitor.onCommand(InactivityMonitor.java:138)
      at org.apache.activemq.transport.TransportSupport.doConsume(TransportSupport.java:83)
      at org.apache.activemq.transport.tcp.TcpTransport.doRun(TcpTransport.java:185)
      at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:172)
      at java.lang.Thread.run(Thread.java:619)
      .........................ignore the more exception stack by me

        Activity

        Hide
        qingwu lin added a comment -

        I deploy brokers clustering topology as pure master/slave.

        Show
        qingwu lin added a comment - I deploy brokers clustering topology as pure master/slave.
        Hide
        Torsten Mielke added a comment -

        I have noticed the same.
        It was not even necessary to kill the slave, the following procedure reproduced the same behaviour using this sequence of steps:

        • start the master
        • start the slave
        • stop the slave gracefully (CTRL+C in slave console)
        • try to browse any of the master's queues. It will raise the above exception in the browser window.

        We also connected JMX console to the master broker after the slave's shutdown and noticed that JMX did not show any queues anymore. However, under the Brokers MBean properties, the Queues still show up in the property called "Queues" of type javax.management.ObjectName[].

        We tested using 5.3.0.5-fuse.

        This kind of makes pure master/slave unusable IMHO.

        Show
        Torsten Mielke added a comment - I have noticed the same. It was not even necessary to kill the slave, the following procedure reproduced the same behaviour using this sequence of steps: start the master start the slave stop the slave gracefully (CTRL+C in slave console) try to browse any of the master's queues. It will raise the above exception in the browser window. We also connected JMX console to the master broker after the slave's shutdown and noticed that JMX did not show any queues anymore. However, under the Brokers MBean properties, the Queues still show up in the property called "Queues" of type javax.management.ObjectName[]. We tested using 5.3.0.5-fuse. This kind of makes pure master/slave unusable IMHO.
        Hide
        Gary Tully added a comment -

        Does trunk or a 5.4-SNAPSHOT behave the same?

        Show
        Gary Tully added a comment - Does trunk or a 5.4-SNAPSHOT behave the same?
        Hide
        Oleg Kiorsak added a comment -

        I had the same problem.

        The fact that Master halts in case of Slave failure seriously cripples the usefulness of "pure master/slave" which otherwise might been a suitable option for some solutions...

        Agree with Torsten Mielke comment:
        "This kind of makes pure master/slave unusable IMHO"

        Show
        Oleg Kiorsak added a comment - I had the same problem. The fact that Master halts in case of Slave failure seriously cripples the usefulness of "pure master/slave" which otherwise might been a suitable option for some solutions... Agree with Torsten Mielke comment: "This kind of makes pure master/slave unusable IMHO"
        Hide
        Gary Tully added a comment -

        There is a shutdownOnSlaveFailure attribute on the broker that will ensure that a master dies when the slave dies, both need to be restarted to continue.

        Show
        Gary Tully added a comment - There is a shutdownOnSlaveFailure attribute on the broker that will ensure that a master dies when the slave dies, both need to be restarted to continue.
        Hide
        Timothy Bish added a comment -

        Pure master/slave removed in upcoming v5.8.0

        Show
        Timothy Bish added a comment - Pure master/slave removed in upcoming v5.8.0

          People

          • Assignee:
            Unassigned
            Reporter:
            qingwu lin
          • Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development