Uploaded image for project: 'ActiveMQ Classic'
  1. ActiveMQ Classic
  2. AMQ-5274

Stuck messages and CPU churn when aborted transacted message expires

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • 5.8.0, 5.9.0, 5.9.1, 5.10.0
    • 5.11.0
    • None
    • None
    • win64, RHEL11

    Description

      The test case is a simple producer/consumer:

      Producer: 20 messages are injected, with a timeout of 10s.

      Consumer: The redelivery policy is set to 0 retries (the issue exists with other values). The consumer uses transactions and throws a runtime exception on each message received.

      queue stats show 20 enqueue, 19 dequeue, 1 pending

      DLQ stat show 20 enqueue: all 20 messages go to DLQ, IDs ending in 1-10 for failing, 11-20 for expiry (approx)

      the pending item (ID ending in 10) is a "ghost message" , and remains stuck indefinitely in queue.test

      if you browse, the message is not shown

      A) if you restart the broker, after a short while the message is cleaned:

      jvm 1 | WARN | Duplicate message add attempt rejected. Destination: QUEUE://ActiveMQ.DLQ, Message id: ID:REDACTED-52872-1405079629779-1:1:1:1:10
      jvm 1 | WARN | org.apache.activemq.broker.region.cursors.QueueStorePrefetch@5b427f3c:ActiveMQ.DLQ,batchResetNeeded=false,storeHasMessages=true,size=0,cacheEnabled=true,maxBatchSize:20,hasSpace:tru
      e - cursor got duplicate: ID:REDACTED--52872-1405079629779-1:1:1:1:10, 4
      jvm 1 | WARN | duplicate message from store ID:REDACTED--52872-1405079629779-1:1:1:1:10, redirecting for dlq processing

      B) if you purge, ActiveMQ logs a warning: "WARN | queue://queue.test after purge complete, message count stats report: 1"
      the queue is marked as being empty.
      however if you restart the broker, the message re-appears shorty, before being cleaned as above

      SUPPLEMENTARY: with activeMQ 5.9.0 and above , if you run the injection several times, the CPU usage of ActiveMQ climbs drastically until the queue is purged.

      Attachments

        1. logs_extract.txt
          3 kB
          Yannick Malins
        2. AMQ-5274v2.zip
          11 kB
          Torsten Mielke
        3. AMQ5274Test.java
          6 kB
          Timothy A. Bish
        4. AMQ-5274.zip
          7 kB
          Yannick Malins

        Issue Links

          Activity

            People

              gtully Gary Tully
              yannick.malins Yannick Malins
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: