Details
Major Description
Sometimes off and on the ActiveMQ server, the KahaDB maintains some old transactions that try to recovery. So at start up these transactions are added to the xaTransactions map inside TransactionBroker with a null ConnectionId.
This is the stack trace of the recovery at startup:
TransactionBroker.beginTransaction(ConnectionContext, TransactionId) line: 152 TransactionBroker$1.recover(XATransactionId, Message[], MessageAck[]) line: 92 KahaDBTransactionStore.recover(TransactionRecoveryListener) line: 317 TransactionBroker.start() line: 89 BrokerService$3.start() line: 1781 XBeanBrokerService(BrokerService).start() line: 489 XBeanBrokerService.afterPropertiesSet() line: 60 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 597 DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeCustomInitMethod(String, Object, RootBeanDefinition) line: 1536 DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).invokeInitMethods(String, Object, RootBeanDefinition) line: 1477 DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).initializeBean(String, Object, RootBeanDefinition) line: 1409 DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).doCreateBean(String, RootBeanDefinition, Object[]) line: 519 DefaultListableBeanFactory(AbstractAutowireCapableBeanFactory).createBean(String, RootBeanDefinition, Object[]) line: 456 AbstractBeanFactory$1.getObject() line: 291 DefaultListableBeanFactory(DefaultSingletonBeanRegistry).getSingleton(String, ObjectFactory) line: 222 DefaultListableBeanFactory(AbstractBeanFactory).doGetBean(String, Class<T>, Object[], boolean) line: 288 DefaultListableBeanFactory(AbstractBeanFactory).getBean(String) line: 190 DefaultListableBeanFactory.preInstantiateSingletons() line: 574 XBeanBrokerFactory$1(AbstractApplicationContext).finishBeanFactoryInitialization(ConfigurableListableBeanFactory) line: 895 XBeanBrokerFactory$1(AbstractApplicationContext).refresh() line: 425 XBeanBrokerFactory$1(ResourceXmlApplicationContext).<init>(Resource, List) line: 64 XBeanBrokerFactory$1(ResourceXmlApplicationContext).<init>(Resource) line: 52 XBeanBrokerFactory$1.<init>(XBeanBrokerFactory, Resource) line: 115 XBeanBrokerFactory.createApplicationContext(String) line: 115 XBeanBrokerFactory.createBroker(URI) line: 71 BrokerFactory.createBroker(URI, boolean) line: 71 BrokerFactory.createBroker(URI) line: 54 StartCommand.startBroker(URI) line: 115 StartCommand.runTask(List<String>) line: 74 StartCommand(AbstractCommand).execute(List<String>) line: 57 ShellCommand.runTask(List<String>) line: 143 ShellCommand(AbstractCommand).execute(List<String>) line: 57 ShellCommand.main(String[], InputStream, PrintStream) line: 85 NativeMethodAccessorImpl.invoke0(Method, Object, Object[]) line: not available [native method] NativeMethodAccessorImpl.invoke(Object, Object[]) line: 39 DelegatingMethodAccessorImpl.invoke(Object, Object[]) line: 25 Method.invoke(Object, Object...) line: 597 Main.runTaskClass(List<String>) line: 251 Main.main(String[]) line: 107
During the runtime the client tries to add and remove connections; sometimes the removeConnection throws a NPE due to these transactions without ConnectionID.
Take a look to the code fragment from TransactionBroker:
public void removeConnection(ConnectionContext context, ConnectionInfo info, Throwable error) throws Exception { for (Iterator<Transaction> iter = context.getTransactions().values().iterator(); iter.hasNext();) { try { Transaction transaction = iter.next(); transaction.rollback(); } catch (Exception e) { LOG.warn("ERROR Rolling back disconnected client's transactions: ", e); } iter.remove(); } synchronized (xaTransactions) { // first find all txs that belongs to the connection ArrayList<XATransaction> txs = new ArrayList<XATransaction>(); for (XATransaction tx : xaTransactions.values()) { if (tx.getConnectionId().equals(info.getConnectionId()) && !tx.isPrepared()) { txs.add(tx); } } // then remove them // two steps needed to avoid ConcurrentModificationException, from removeTransaction() for (XATransaction tx : txs) { try { tx.rollback(); } catch (Exception e) { LOG.warn("ERROR Rolling back disconnected client's xa transactions: ", e); } } } next.removeConnection(context, info, error); }
as you can see inside the loop there is a check for tx.getConnectionId().equals(info.getConnectionId()) that throws the NPE. When this occurs the connection isn't removed. This information isn't shared with the client that believes the opposite, so the next time that try to resend client information to server obtain (under jBoss) this error javax.transaction.xa.XAException: Broker: AMQ - Client: ID:srv001-47592-1336730655955-64:2 already connected from /127.0.0.1:49806
that can be bound to the former server fails.
This scenario can be found inside the attached logs.