Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-5274

Stuck messages and CPU churn when aborted transacted message expires

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: 5.8.0, 5.9.0, 5.9.1, 5.10.0
    • Fix Version/s: 5.11.0
    • Component/s: None
    • Labels:
      None
    • Environment:

      win64, RHEL11

      Description

      The test case is a simple producer/consumer:

      Producer: 20 messages are injected, with a timeout of 10s.

      Consumer: The redelivery policy is set to 0 retries (the issue exists with other values). The consumer uses transactions and throws a runtime exception on each message received.

      queue stats show 20 enqueue, 19 dequeue, 1 pending

      DLQ stat show 20 enqueue: all 20 messages go to DLQ, IDs ending in 1-10 for failing, 11-20 for expiry (approx)

      the pending item (ID ending in 10) is a "ghost message" , and remains stuck indefinitely in queue.test

      if you browse, the message is not shown

      A) if you restart the broker, after a short while the message is cleaned:

      jvm 1 | WARN | Duplicate message add attempt rejected. Destination: QUEUE://ActiveMQ.DLQ, Message id: ID:REDACTED-52872-1405079629779-1:1:1:1:10
      jvm 1 | WARN | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@5b427f3c:ActiveMQ.DLQ,batchResetNeeded=false,storeHasMessages=true,size=0,cacheEnabled=true,maxBatchSize:20,hasSpace:tru
      e - cursor got duplicate: ID:REDACTED--52872-1405079629779-1:1:1:1:10, 4
      jvm 1 | WARN | duplicate message from store ID:REDACTED--52872-1405079629779-1:1:1:1:10, redirecting for dlq processing

      B) if you purge, ActiveMQ logs a warning: "WARN | queue://queue.test after purge complete, message count stats report: 1"
      the queue is marked as being empty.
      however if you restart the broker, the message re-appears shorty, before being cleaned as above

      SUPPLEMENTARY: with activeMQ 5.9.0 and above , if you run the injection several times, the CPU usage of ActiveMQ climbs drastically until the queue is purged.

        Attachments

        1. AMQ-5274.zip
          7 kB
          Yannick Malins
        2. logs_extract.txt
          3 kB
          Yannick Malins
        3. AMQ5274Test.java
          6 kB
          Timothy A. Bish
        4. AMQ-5274v2.zip
          11 kB
          Torsten Mielke

          Issue Links

            Activity

              People

              • Assignee:
                gtully Gary Tully
                Reporter:
                yannick.malins Yannick Malins
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: