[ARTEMIS-699] Reaper seems to clog up the journal, and messages fail to deliver

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.4.0
• Component/s: Broker
• Assignee: Justin Bertram
• Reporter: Michael Brown
• Created: 25/Aug/16 11:23
• Updated: 03/Nov/16 20:40
• Resolved: 02/Sep/16 13:47

Description

We are seeing the exception (below) regularly, and a loss of message delivery, which only goes away if we delete the journal and restart. This is with the following grouping options in broker.xml:

    <grouping-handler name="jms-recordsure-grouping-handler">
      <type>LOCAL</type>
      <address>jms</address>
      <timeout>5000</timeout>
      <group-timeout>300000</group-timeout>
      <reaper-period>30000</reaper-period>
    </grouping-handler>

If we remove timeout/group-timeout/reaper-period the issues go away.

Having traced through the code I see this issue in LocalGroupHandling.java:

                     if (expiredGroups >= 1000 && txID >= 0) {
                        expiredGroups = 0;
                        txID = -1;
                        storageManager.commitBindings(txID);
                     }

The method 'commitBindings' has (via dependencies) an explicit check that will throw an exception if txID is -1. This code must fail.

I suggest a fix that moves the reset of txID to be after the commitBindings call, as follows:

                     if (expiredGroups >= 1000 && txID >= 0) {
                        storageManager.commitBindings(txID);
                        expiredGroups = 0;
                        txID = -1;
                     }

2016-08-24 13:09:01,660 WARN  [org.apache.activemq.artemis.core.server] AMQ222114: Unable to delete group binding info c2e95857-87ad-455a-ba67-1c63ac351cc8.ad900ba9-69e3-11e6-86b2-0242ac110003: java.lang.IllegalStateException: Cannot find tx with id -1
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendCommitRecord(JournalImpl.java:1013) [artemis-journal-1.3.0.jar:1.3.0]
        at org.apache.activemq.artemis.core.journal.impl.JournalBase.appendCommitRecord(JournalBase.java:112) [artemis-journal-1.3.0.jar:1.3.0]
        at org.apache.activemq.artemis.core.journal.impl.JournalImpl.appendCommitRecord(JournalImpl.java:79) [artemis-journal-1.3.0.jar:1.3.0]
        at org.apache.activemq.artemis.core.replication.ReplicatedJournal.appendCommitRecord(ReplicatedJournal.java:164) [artemis-server-1.3.0.jar:1.3.0]
        at org.apache.activemq.artemis.core.persistence.impl.journal.AbstractJournalStorageManager.commitBindings(AbstractJournalStorageManager.java:650) [artemis-server-1.3.0.jar:1.3.0]
        at org.apache.activemq.artemis.core.server.group.impl.LocalGroupingHandler$GroupIdReaper.run(LocalGroupingHandler.java:449) [artemis-server-1.3.0.jar:1.3.0]
        at org.apache.activemq.artemis.utils.OrderedExecutorFactory$OrderedExecutor$ExecutorTask.run(OrderedExecutorFactory.java:103) [artemis-core-client-1.3.0.jar:1.3.0]
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [rt.jar:1.8.0_102]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [rt.jar:1.8.0_102]
        at java.lang.Thread.run(Thread.java:745) [rt.jar:1.8.0_102]

Issue Links

• duplicates: ARTEMIS-687 AMQ222114: Unable to delete group binding info <groupid.messageId>: java.lang.IllegalStateException: Cannot find tx with id -1
• is duplicated by: ARTEMIS-687 AMQ222114: Unable to delete group binding info <groupid.messageId>: java.lang.IllegalStateException: Cannot find tx with id -1
• links to: GitHub Pull Request #743