ActiveMQ
  1. ActiveMQ
  2. AMQ-1786

Journal files don't get cleaned up

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 5.1.0
    • Fix Version/s: 5.3.0
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      Linux server
      Sun JDK 1.6.0

      Description

      I'm running ActiveMQ 5.1.0 with the AMQ persistence adapter, and it appears that not all of the journal files get cleaned up. My setup is a little abnormal, as I'm trying to test out ActiveMQ's ability to handle queue messaging with consumers that may become inactive for periods of time.

      So for this test, I have a single publisher pushing messages out to 21 queues. These are persistent messages with an expiration time. I have a message listener reading from all queues (reading from '>'). So, as soon as the message is sent to the queues, it's read by the message listener, taking it off the queue. So far, so good.

      I have a 2 MB max file length set on the AMQ persistence adapter. So, I would expect to see for the journal, 2 MB files that get cleaned up after the file rolls over. However, the journal files don't always get cleaned up, as shown in the file listing below. Out of 181 rollovers, 30 of the files did not get cleaned up. The message listener showed no errors, and as far as I can tell, it didn't drop any messages.

      -rw-r--r-- 1 root root  2096753 2008-05-30 20:30 data/journal/data-13
      -rw-r--r-- 1 root root  2096967 2008-05-30 20:31 data/journal/data-14
      -rw-r--r-- 1 root root  2096899 2008-05-30 20:45 data/journal/data-25
      -rw-r--r-- 1 root root  2097057 2008-05-30 21:20 data/journal/data-52
      -rw-r--r-- 1 root root  2096916 2008-05-30 21:22 data/journal/data-54
      -rw-r--r-- 1 root root  2096536 2008-05-30 21:45 data/journal/data-72
      -rw-r--r-- 1 root root  2096894 2008-05-30 21:47 data/journal/data-73
      -rw-r--r-- 1 root root  2097129 2008-05-30 21:49 data/journal/data-75
      -rw-r--r-- 1 root root  2097101 2008-05-30 21:58 data/journal/data-82
      -rw-r--r-- 1 root root  2097026 2008-05-30 21:59 data/journal/data-83
      -rw-r--r-- 1 root root  2096906 2008-05-30 22:02 data/journal/data-85
      -rw-r--r-- 1 root root  2096973 2008-05-30 22:13 data/journal/data-94
      -rw-r--r-- 1 root root  2097105 2008-05-30 22:24 data/journal/data-102
      -rw-r--r-- 1 root root  2097033 2008-05-30 22:41 data/journal/data-113
      -rw-r--r-- 1 root root  2096730 2008-05-30 22:42 data/journal/data-114
      -rw-r--r-- 1 root root  2096569 2008-05-30 22:45 data/journal/data-116
      -rw-r--r-- 1 root root  2096870 2008-05-30 22:50 data/journal/data-118
      -rw-r--r-- 1 root root  2096567 2008-05-30 22:52 data/journal/data-119
      -rw-r--r-- 1 root root  2096766 2008-05-30 23:03 data/journal/data-128
      -rw-r--r-- 1 root root  2096877 2008-05-30 23:06 data/journal/data-130
      -rw-r--r-- 1 root root  2096888 2008-05-30 23:18 data/journal/data-140
      -rw-r--r-- 1 root root  2096699 2008-05-30 23:20 data/journal/data-141
      -rw-r--r-- 1 root root  2096973 2008-05-30 23:22 data/journal/data-143
      -rw-r--r-- 1 root root  2096924 2008-05-30 23:31 data/journal/data-150
      -rw-r--r-- 1 root root  2096936 2008-05-30 23:45 data/journal/data-161
      -rw-r--r-- 1 root root  2096527 2008-05-30 23:57 data/journal/data-170
      -rw-r--r-- 1 root root  2097151 2008-05-30 23:58 data/journal/data-171
      -rw-r--r-- 1 root root  2096972 2008-05-31 00:11 data/journal/data-179
      -rw-r--r-- 1 root root  2096703 2008-05-31 00:13 data/journal/data-180
      -rw-r--r-- 1 root root  2097069 2008-05-31 00:14 data/journal/data-181
      

      I've also tried taking out the wildcard '>' on a single consumer, and instead used separate consumers for each queue, and I get the same result.

      I haven't even gotten to the test yet where the listener is not running. So, in this "normal" operation, all messages are consumed. Yet, not all journal files get cleaned up. These left-over files don't ever get cleaned up. They will eventually start filling the hard drive. I can understand files being left behind when there's no consumer, but there is a consumer the whole time.

      What I'm basically looking for is a persistence layer for messaging to multiple clients, so that consumers can get messages retroactively when they start up. I could try to use topics with durable clients, but I thought the queues would be easier to setup, as messages in queues are persisted by default. However, I don't want the consumer to process "stale" messages, which is why I set an expiration time. So, I would think that, with a constant rate of messages, the persistent disk store utilization would eventually level out as the messages started to expire. I realize that if there's no consumer for a queue, expired messages won't get cleaned up (am currently trying to figure out a work-around for that - periodically checking the queues with a QueueBrowser seems to trigger the removal of expired messages). However, even when all consumers are active, the journal keeps growing, as it's not always cleaning up it's files!

      I've attached my configuration to this ticket.

        Activity

        Hide
        James C added a comment -

        I'm on vacation until Monday 24th August 2009.
        I will read your mail on return.
        Thanks,
        James Casey.

        Show
        James C added a comment - I'm on vacation until Monday 24th August 2009. I will read your mail on return. Thanks, James Casey.
        Hide
        Rob Davies added a comment -

        With https://issues.apache.org/activemq/browse/AMQ-1112 now resolved - to actively purge expired messages from the store - this is issue is resolved

        Show
        Rob Davies added a comment - With https://issues.apache.org/activemq/browse/AMQ-1112 now resolved - to actively purge expired messages from the store - this is issue is resolved
        Hide
        Gary Tully added a comment -

        sure, lots of folks uses it. For best performance use it with the Journal. ie: journaledJDBC

        Show
        Gary Tully added a comment - sure, lots of folks uses it. For best performance use it with the Journal. ie: journaledJDBC
        Hide
        elliot barlas added a comment -

        Gary, for those of us who are concerned about the inefficient use of log files, do you think the jdbc persistence adapter is a suitable alternative?

        Show
        elliot barlas added a comment - Gary, for those of us who are concerned about the inefficient use of log files, do you think the jdbc persistence adapter is a suitable alternative?
        Hide
        Gary Tully added a comment -

        one option for you may be to reduce the size of the data files, there will be more of them but the chances of them having unconsumed messages will be reduced. also, increase the checkpoint and cleanup interval.

        Show
        Gary Tully added a comment - one option for you may be to reduce the size of the data files, there will be more of them but the chances of them having unconsumed messages will be reduced. also, increase the checkpoint and cleanup interval.
        Hide
        Sudip Pulapura added a comment -

        Thanks for the reply. It would be great if you could take the above comments into account when looking at enhancements for activemq. It would be a good improvement to not have entire log files held up by a few unconsumed messages.

        Show
        Sudip Pulapura added a comment - Thanks for the reply. It would be great if you could take the above comments into account when looking at enhancements for activemq. It would be a good improvement to not have entire log files held up by a few unconsumed messages.
        Hide
        Gary Tully added a comment -

        Sudip,
        this is part of the contract when you make delivery persistent. The store cannot discard the message. Expiry with a consumer is your only option till an expiry processor can be added to activemq. see: https://issues.apache.org/activemq/browse/AMQ-1112

        Show
        Gary Tully added a comment - Sudip, this is part of the contract when you make delivery persistent. The store cannot discard the message. Expiry with a consumer is your only option till an expiry processor can be added to activemq. see: https://issues.apache.org/activemq/browse/AMQ-1112
        Hide
        Sudip Pulapura added a comment -

        The class takes the persistent directory as an argument.., the broker is named brokerOne, for ex: <basedir>/activemq-data/brokerOne/journal

        The class starts up a broker, creates a queue "consume.queue" with a normal consumer and producer. It also tries to produce to another queue "no.consume.queue" without a consumer whenever it finds a new log file has been added to the journal directory. The stats will show that adding just a single unconsumed message to each log file prevents it from being cleaned up.

        The issue seems to be inherent in the way persistent data files are handled in activemq. The problem is reproducible in the 5.3-SNAPSHOT too. While a fix is great, any workaround for 5.2.0 will be greatly appreciated.

        Show
        Sudip Pulapura added a comment - The class takes the persistent directory as an argument.., the broker is named brokerOne, for ex: <basedir>/activemq-data/brokerOne/journal The class starts up a broker, creates a queue "consume.queue" with a normal consumer and producer. It also tries to produce to another queue "no.consume.queue" without a consumer whenever it finds a new log file has been added to the journal directory. The stats will show that adding just a single unconsumed message to each log file prevents it from being cleaned up. The issue seems to be inherent in the way persistent data files are handled in activemq. The problem is reproducible in the 5.3-SNAPSHOT too. While a fix is great, any workaround for 5.2.0 will be greatly appreciated.
        Hide
        Gary Tully added a comment -

        Sudip, have you tried a 5.3-SNAPSHOT?
        Could you possibly submit your test case for this scenario ?

        Show
        Gary Tully added a comment - Sudip, have you tried a 5.3-SNAPSHOT? Could you possibly submit your test case for this scenario ?
        Hide
        Sudip Pulapura added a comment -

        We are using ActiveMQ 5.2.0 and see the same problem mentioned in this issue. It looks like the culprit is "with consumers that may become inactive for periods of time. " From the documentation, it does seem that the store clears out a log file only if it finds that there is no reference to a single message in the log file. However, with consumers that go up and down, it is possible that every log file ends up having a single persistent message that is not consumed. Hence, with a message store size of 1gb and a data file size of 32 MB, it takes around 30 messages to kill persistence, which is a very realistic scenario. Is there any method by which the files are consolidated and not held up by a single unconsumed message? Expiring the messages doesn't really work since there needs to be a consumer for messages to expire.

        Show
        Sudip Pulapura added a comment - We are using ActiveMQ 5.2.0 and see the same problem mentioned in this issue. It looks like the culprit is "with consumers that may become inactive for periods of time. " From the documentation, it does seem that the store clears out a log file only if it finds that there is no reference to a single message in the log file. However, with consumers that go up and down, it is possible that every log file ends up having a single persistent message that is not consumed. Hence, with a message store size of 1gb and a data file size of 32 MB, it takes around 30 messages to kill persistence, which is a very realistic scenario. Is there any method by which the files are consolidated and not held up by a single unconsumed message? Expiring the messages doesn't really work since there needs to be a consumer for messages to expire.
        Hide
        Sudip Pulapura added a comment -

        We are using ActiveMQ 5.2.0 and see the same problem mentioned in this issue. It looks like the culprit is "with consumers that may become inactive for periods of time. " From the documentation, it does seem that the store clears out a log file only if it finds that there is no reference to a single message in the log file. However, with consumers that go up and down, it is possible that every log file ends up having a single persistent message that is not consumed. Hence, with a message store size of 1gb and a data file size of 32 MB, it takes around 30 messages to kill persistence, which is a very realistic scenario. Is there any method by which the files are consolidated and not held up by a single unconsumed message? Expiring the messages doesn't really work since there needs to be a consumer for messages to expire.

        Show
        Sudip Pulapura added a comment - We are using ActiveMQ 5.2.0 and see the same problem mentioned in this issue. It looks like the culprit is "with consumers that may become inactive for periods of time. " From the documentation, it does seem that the store clears out a log file only if it finds that there is no reference to a single message in the log file. However, with consumers that go up and down, it is possible that every log file ends up having a single persistent message that is not consumed. Hence, with a message store size of 1gb and a data file size of 32 MB, it takes around 30 messages to kill persistence, which is a very realistic scenario. Is there any method by which the files are consolidated and not held up by a single unconsumed message? Expiring the messages doesn't really work since there needs to be a consumer for messages to expire.
        Hide
        Rob Davies added a comment -

        Slightly different problem - but this was resolved by https://issues.apache.org/activemq/browse/AMQ-1795

        Show
        Rob Davies added a comment - Slightly different problem - but this was resolved by https://issues.apache.org/activemq/browse/AMQ-1795

          People

          • Assignee:
            Rob Davies
            Reporter:
            Brian Desai
          • Votes:
            10 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development