Messages keep hanging in JDBCStore without delivery to client

• Key: AMQ-2184
• Type: Bug
• Status: Closed
• Resolution: Won't Fix
• Priority: Trivial
• Assignee: Rob Davies
• Reporter: Norbert Pfistner
• Component/s: Broker
• Affects Version/s: 5.1.0, 5.2.0
• Fix Version/s: 5.1.0

Environment

OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
ActiveMQ: 5.1.0
1 Broker, 12 JBoss's with round about 20 consumers and 20 Producers on 7 JDBC persistant queues
Clients use transacted consumers.

Description

We currently suffer from hanging messages in the JDBC persistant storage in our production environment.

Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.

Our system depends on processing every message, so missing even one of them is realy a big problem in production.

Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).

Our messaging characteristics show moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.

Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.

The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:

public final synchronized void remove() {
size--;
if (size==0 && isStarted() && cacheEnabled) { cacheEnabled=true; }
}

As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.

Maybe this causes some strange effects when using the StoreCursor's derived from this class..

Comments

[Norbert Pfistner]

Our testing system now also shows this strange behaviour.
The Webmonitor lists correctly the 25 pending messages.
Browsing the lists shows them to be empty – which is obviously not true,
as revealed by a direct select to the database storage in table 'activemq_msgs'.

I attached the screenshots from the web monitor and the db select.

The broker simply forgets some messages.
The only workaround seems to constantly reboot the broker, which is really annoying
and not the kind of thing we'd like to do in a productive environment.

This issue is really is realy urgent!

[Norbert Pfistner]

I also added the JConsole for the IMPORT_READY queue.

It shows the correct queue size of 1, but the message ist still stuck in the queue.

[Norbert Pfistner]

Problem solved:

Eventually we started not only one but actually four brokers.
All of them had the same datasource in the DB.
The main mistake was that we had the following setting which prevented the exclusive DB Table lock for a valid master/slave configuration:

<bean id="persistenceAdapter" class="org.apache.activemq.store.jdbc.JDBCPersistenceAdapter">
<property name="dataSource" ref="brokerDataSource"/>
<property name="useDatabaseLock" value="false"/>
</bean>

After shutting down the additional brokers we do not experience hanging massages any more.

[Rob Davies]

This looks fixed now - thanks Norbert

[Norbert Pfistner]

Unfortunately we are experiencing the prblem again – now with a single standalone broker.

The hanging mesages are listed in the database but not in the visible in the webconsole (browse).

Restarting the connector (using JMX) seems to to be a hotfix (but obvoiously not a solution to the problem).

The listeners wait receiving (as stated by a thread dump), so it seems not to be a client code problem.

BTW: Still using activemq 5.1

http://issues.apache.org/activemq/browse/AMQ-1918 may be related to this problem.

[Rob Davies]

Hi Norbert - we don't do patches to the 5.1 branch - is this an issue with 5.3 (trunk currently) ? If not can you close it ?

[Norbert Pfistner]

As long as we can see it's an issue somehow related to the database persistance. We still did not get any pin on it.
Currently we use 5.1, so as Dave mentioned in the comment above I'll close it.

[Norbert Pfistner]

It seems that we do not get any clear solution for this bug for version 5.1.