[AMQ-2184] Messages keep hanging in JDBCStore without delivery to client

• Type: Bug
• Status: Closed
• Priority: Trivial
• Resolution: Won't Fix
• Affects Version/s: 5.1.0, 5.2.0
• Fix Version/s: 5.1.0
• Component/s: Broker

Environment

OS: Linux version 2.6.26-amd64-85 (root@etch64) (gcc version 4.1.2 20061115 (prerelease) (Debian 4.1.1-21))
ActiveMQ: 5.1.0
1 Broker, 12 JBoss's with round about 20 consumers and 20 Producers on 7 JDBC persistant queues
Clients use transacted consumers.

• Assignee: Robert Davies
• Reporter: Norbert Pfistner
• Created: 26/Mar/09 14:12
• Updated: 07/Jul/09 09:13
• Resolved: 07/Jul/09 09:13

Description

We currently suffer from hanging messages in the JDBC persistant storage in our production environment.

Occasionally (once in a day or week) a few Messages (most time 1 or 2, sometimes more) seems to stuck in the DB Table activemq_msg without being delivered even once to a consumer. They are never deliverd to a consumer until the broker is rebooted. Strange enough new messages produced into the same queue with those stuck messages are delivered to the consumers as normal. It simply looks like the borker is blind about the stuck messages.

Our system depends on processing every message, so missing even one of them is realy a big problem in production.

Everything seems to be fine: No warnings, errors or even debug logging gives any hint about a problem.
Unfortunately we can't debug the broker 'cuse it's our production environment. Our Testing envirenment is not capable to reproduce the problem (actually it's only one Machine with a Broker and a signle jaboss running a small number of producers/clients).

Our messaging characteristics show moderate producers and slow consumers. The count of simultanous messages is moderate (about 300-1000). Actually the producers have hot phases generating bulks of Messages followed by phases of long inactivity. The consumers are slow related to the producers but always keep within an hour or two. We do not experience any memory problems.

Because this bug may be related to https://issues.apache.org/activemq/browse/AMQ-1918, I had a look at the code around the StoreCursors and stumbled apon a strange code fragment called when removing entries.

The suspicous code is found in org.apache.activemq.broker.region.cursors.AbstractStoreCursor.remove() in 5.1.0, 5.2 .0 and 5.3.0:

public final synchronized void remove() {
size--;
if (size==0 && isStarted() && cacheEnabled)

{ cacheEnabled=true; }

}

As far as i can see, the if statement ist actually a NOP, 'cause it will not modify cacheEnabled in any case.

Maybe this causes some strange effects when using the StoreCursor's derived from this class..