IllegalArgumentException-Subscripiton does not exist when client disconnects

• Key: AMQ-1711
• Type: Bug
• Status: Open
• Priority: Major
• Assignee: Unassigned
• Reporter: Howard Orner
• Affects Version/s: 4.1.1

Description

When a client disconnects in a Network of Brokers configuration, it is possible for an IllegalArgumentException to be thrown from org.apache.activemq.broker.region.AbstractRegion.acknowlege() due to a race condition between message processing and disconnect subscription clean-up. It is easily reproducable by putting two brokers in a Network of Brokers configuration (We will call them A and B). If you publish message at a high rate to broker A (I found that if I published messages at the rate of 1 every 10ms this happened consistantly – better hardware may require a high rate so I recommend pushing message as fast as possible), put a subscriber on broker B (my subscriber used selectors but I don't think that makes a difference), and kill the subscriber process (I don't know if a friendly disconnect behaves better), Broker A will throw an exception and disconnect broker B perminately from the NOB configuration. All future connections to broker B will not received messages published to A.

I found that if I changed AbstractRegion.acknowledge to log the error, but not throw an exception that it solved the problem with seemly no ill effects.

Comments

[Kai Hudalla]

I am experiencing the same problem during recovery of a previously failing broker. When I kill a broker that I am sending messages to continuously and then re-start the broker, I get the following stack trace during startup and the client cannot re-connect to the broker:

ERROR: java.lang.Exception: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.act
ivemq.xbean.XBeanBrokerService' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception
is java.io.IOException: Recovery Failed: The subscription does not exist: ID:FE-Z2965-3182-1210069658579-1:567:-1:2
java.lang.Exception: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.x
bean.XBeanBrokerService' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java
.io.IOException: Recovery Failed: The subscription does not exist: ID:FE-Z2965-3182-1210069658579-1:567:-1:2
at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:71)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:50)
at org.apache.activemq.console.command.ShellCommand.runTask(ShellCommand.java:65)
at org.apache.activemq.console.command.AbstractCommand.execute(AbstractCommand.java:50)
at org.apache.activemq.console.command.ShellCommand.main(ShellCommand.java:46)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.activemq.console.Main.runTaskClass(Main.java:202)
at org.apache.activemq.console.Main.main(Main.java:91)
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'org.apache.activemq.xbean.XBean
BrokerService' defined in class path resource [activemq.xml]: Invocation of init method failed; nested exception is java.io.IOExce
ption: Recovery Failed: The subscription does not exist: ID:FE-Z2965-3182-1210069658579-1:567:-1:2
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBean
Factory.java:1032)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFact
ory.java:420)
at org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:245)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:1
41)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:242)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:156)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactor
y.java:290)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:348)
at org.apache.xbean.spring.context.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:161)
at org.apache.xbean.spring.context.ClassPathXmlApplicationContext.<init>(ClassPathXmlApplicationContext.java:51)
at org.apache.activemq.xbean.XBeanBrokerFactory.createBroker(XBeanBrokerFactory.java:41)
at org.apache.activemq.broker.BrokerFactory.createBroker(BrokerFactory.java:57)
at org.apache.activemq.console.command.StartCommand.startBroker(StartCommand.java:82)
at org.apache.activemq.console.command.StartCommand.runTask(StartCommand.java:47)
... 10 more
Caused by: java.io.IOException: Recovery Failed: The subscription does not exist: ID:FE-Z2965-3182-1210069658579-1:567:-1:2
at org.apache.activemq.util.IOExceptionSupport.create(IOExceptionSupport.java:25)
at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:100)
at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
at org.apache.activemq.broker.BrokerFilter.start(BrokerFilter.java:154)
at org.apache.activemq.broker.MutableBrokerFilter.start(MutableBrokerFilter.java:164)
at org.apache.activemq.broker.BrokerService.start(BrokerService.java:394)
at org.apache.activemq.xbean.XBeanBrokerService.afterPropertiesSet(XBeanBrokerService.java:47)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableB
eanFactory.java:1062)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBean
Factory.java:1029)
... 23 more
Caused by: java.lang.IllegalArgumentException: The subscription does not exist: ID:FE-Z2965-3182-1210069658579-1:567:-1:2
at org.apache.activemq.broker.region.AbstractRegion.acknowledge(AbstractRegion.java:298)
at org.apache.activemq.broker.region.RegionBroker.acknowledge(RegionBroker.java:402)
at org.apache.activemq.broker.TransactionBroker.acknowledge(TransactionBroker.java:177)
at org.apache.activemq.broker.TransactionBroker$1.recover(TransactionBroker.java:90)
at org.apache.activemq.store.journal.JournalTransactionStore.recover(JournalTransactionStore.java:240)
at org.apache.activemq.broker.TransactionBroker.start(TransactionBroker.java:82)
... 30 more
INFO ManagementContext - JMX consoles can connect to service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

Kai