Uploaded image for project: 'ActiveMQ'
  1. ActiveMQ
  2. AMQ-6354

KahaDb journal file piling up because of Durable subscriber destination

    XMLWordPrintableJSON

    Details

    • Flags:
      Important

      Description

      AMQ Configuration:
      1. KahaDb Persistence store
      2. Topics and Queues destination are configured.
      3. Durable Subscribers configured (added/removed) dynamically (All added subscribers will be active) for Topic destination.

      Issue:

      • Nearly ~20 Durable Subscriber added/deleted to a particular destination dynamically.
      • After configuration, Once the message are sent to the Topic, the messages are consumed and Acknowledged by each subscribers.
      • There were 249 Subscribers and we could see from Journal file that all subscribers acknowledged.
      • In JMX logs after message is sent the below entries are observed for yyyy destination,
      o Consumer Count is 249
      o In-Flight Count is 0
      o Dispatch count is matching with the consumer count (messages * Consumer count)

      • We enabled kahaDB TRACE logs and it clearly points out that for a particular Topic destination the journal files could not be gc collected

      Snippet of activemq.log

      ----------------------------------------------
      2016-06-30 01:41:43,980 org.apache.activemq.store.kahadb.MessageDatabase TRACE (MessageDatabase.java:checkpointUpdate:1413) [ActiveMQ Journal Checkpoint Worker] [24] | gc candidates after dest:0:xxxxx, [8066, 8067, ...........]

      2016-06-30 01:41:43,980 org.apache.activemq.store.kahadb.MessageDatabase TRACE (MessageDatabase.java:checkpointUpdate:1413) [ActiveMQ Journal Checkpoint Worker] [24] | gc candidates after dest:1:yyyy, [8067, 8073, ............]
      --------------------------------------------

      • After analyzing the 8066 file, we understood that 8066 has one message for yyyy Topic destination and 249 ACk’s in the same file. But Checkpoint cleanup could not remove the file. So this pattern followed and all the files containing the yyyy destination messages are not removed, which leads to journal files pile-up.

      Workaround :

      • As a work around, first we restarted the process but still the files were not cleaned up.
      • Secondly, we stopped the process, removed the index (db.data) file and restarted. Now all the files are cleaned up.
      • By above workaround, we suspect that index file is corrupted, which is causing the journal pile up.
      • Our question,
      o if index file is corrupted, what is the reason for corruption?
      o Is there any bug in the 5.5.1 version of AMQ regarding index file corruption for Durable Subscription?

      This is sporadic issue and not reproducible always. (Not always Dynamic Subscription configuration leads to journal pile up)

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              Rajagopal Rajagopal
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: