Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Incomplete
    • Affects Version/s: 5.5.1
    • Fix Version/s: None
    • Component/s: Broker
    • Labels:
      None
    • Environment:

      cent os 5.3, java 6 update 30

      Description

      In our production and development environment db.data file is growing. Within few weeks it reaches GB's in size. When it grows larger broker hangs frequently. I have already reported a bug in 5.4.2(AMQ-3710) url https://issues.apache.org/jira/browse/AMQ-3710. In that case db.data had a size of 5 GB. When we cleared kaha db related files db.data,db.redo and db-735.log files, that issue solved.By further monitoring We noticed that db.data file is growing by day from MB to GB. And when the size become larger, the broker hanging issue is coming back.

      Now we have updated Our activemq broker to 5.5.1. But db.data is still growing. Every day we are monitoring it and if it is become big(200 mb) we are clearing all kaha related files before our business start.

      Other kaha related files are normal. db-*.log files are cleared by kaha db. db.redo file is not growing. Only db-data file is growing.

      I don't think this is a correct behavior. At certain point of time it should clear unwanted data write. I am using the same configuration without much change.
      I have attached active mq config file with this.

      1. activemq.xml
        6 kB
        Krishnadasan T S

        Activity

        Hide
        Krishnadasan T S added a comment -

        I am attaching activemq.xml file with this.

        Show
        Krishnadasan T S added a comment - I am attaching activemq.xml file with this.
        Hide
        Timothy Bish added a comment -

        If you can provide some test cases that show how you are using the broker when the db.data file grows it would help to diagnose this.

        Show
        Timothy Bish added a comment - If you can provide some test cases that show how you are using the broker when the db.data file grows it would help to diagnose this.
        Hide
        Krishnadasan T S added a comment -

        We are using activemq pool connection library in our client application.Is there any document that explain about how and when the db.data file will be cleared. I will try to make a test case and generate the same issue.

        Show
        Krishnadasan T S added a comment - We are using activemq pool connection library in our client application.Is there any document that explain about how and when the db.data file will be cleared. I will try to make a test case and generate the same issue.
        Hide
        Pat Fox added a comment -

        @Krishnadasan T S, perhaps the following link may be of interest to you http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html.

        Show
        Pat Fox added a comment - @Krishnadasan T S, perhaps the following link may be of interest to you http://activemq.apache.org/why-do-kahadb-log-files-remain-after-cleanup.html .
        Hide
        Krishnadasan T S added a comment -

        in my case db-.log are cleaned as normal. There is no pending .log files(there will be always one file) in kahadb folder. But db-.data file is getting larger and it is never cleaned up. in the above given link it is saying about db-*.log write. But it is not telling about how and when db.data file getting cleared. Can i get some guidance or document about that.

        Show
        Krishnadasan T S added a comment - in my case db- .log are cleaned as normal. There is no pending .log files(there will be always one file) in kahadb folder. But db- .data file is getting larger and it is never cleaned up. in the above given link it is saying about db-*.log write. But it is not telling about how and when db.data file getting cleared. Can i get some guidance or document about that.
        Hide
        Gary Tully added a comment -

        there is no explicit cleanup of db.data, it holds the binary tree index, the expectation is that it will reach a maximum when the maximum depth of a queue is reached or the max number of destinations are created.
        The only solution may be a periodic rebuild of the index through a stop,delete db.data, restart cycle.
        Does your use case require unlimited numbers of destinations or durable consumers or just really large queue depths?

        Show
        Gary Tully added a comment - there is no explicit cleanup of db.data, it holds the binary tree index, the expectation is that it will reach a maximum when the maximum depth of a queue is reached or the max number of destinations are created. The only solution may be a periodic rebuild of the index through a stop,delete db.data, restart cycle. Does your use case require unlimited numbers of destinations or durable consumers or just really large queue depths?
        Hide
        Krishnadasan T S added a comment -

        Hi,

        thanks for a quick reply

        We are using fixed number of queues. Is it applicable to topics also. In the case of topics there is such a scenario like we are creating a topic for each login by appending login id and a newly created session key. so this will be different for each login for a same client. This is not queue but topic. Is this can create the above issue. One thing is that previous topics created for a client will not be used again, so will it be automatically deleted from the binary tree index or I need to explicitly delete it. How to delete topic, i could not see api for delete in jms spec.

        Although now this system is not that large, we really want to scale number of clients. We have a working system, which has nearly 30000 clients base perday, which uses proprietory technologies like Tibco Rendezvous Messaging system,Oracle DB etc. Our goal of this application, which uses open source technologies (activemq, mysql, jboss etc) is to replace that big system with in a 1 year of target. So your support in solving such issues will be very helpful to me.

        Show
        Krishnadasan T S added a comment - Hi, thanks for a quick reply We are using fixed number of queues. Is it applicable to topics also. In the case of topics there is such a scenario like we are creating a topic for each login by appending login id and a newly created session key. so this will be different for each login for a same client. This is not queue but topic. Is this can create the above issue. One thing is that previous topics created for a client will not be used again, so will it be automatically deleted from the binary tree index or I need to explicitly delete it. How to delete topic, i could not see api for delete in jms spec. Although now this system is not that large, we really want to scale number of clients. We have a working system, which has nearly 30000 clients base perday, which uses proprietory technologies like Tibco Rendezvous Messaging system,Oracle DB etc. Our goal of this application, which uses open source technologies (activemq, mysql, jboss etc) is to replace that big system with in a 1 year of target. So your support in solving such issues will be very helpful to me.
        Hide
        Krishnadasan T S added a comment -

        Hi,

        As per documentation it is said that "The metadata store contains the complete broker metadata, consisting mainly of a B-tree index giving the message locations in the data logs". So if the previous data logs files are cleared then this B-tree node corresponding to that file also will be cleared write. So in that case these B-tree will be shrinking and growing periodically. I don't know i am write or wrong. In my case there is only current working data-*.log file (only one file) there, previous files are cleared.

        So can i conclude that the cause for growing db.data is because of the strategy of topic creation logic mentioned in the previous post.

        Show
        Krishnadasan T S added a comment - Hi, As per documentation it is said that "The metadata store contains the complete broker metadata, consisting mainly of a B-tree index giving the message locations in the data logs". So if the previous data logs files are cleared then this B-tree node corresponding to that file also will be cleared write. So in that case these B-tree will be shrinking and growing periodically. I don't know i am write or wrong. In my case there is only current working data-*.log file (only one file) there, previous files are cleared. So can i conclude that the cause for growing db.data is because of the strategy of topic creation logic mentioned in the previous post.
        Hide
        Gary Tully added a comment -

        To get the unused topics cleaned up, see: http://activemq.apache.org/delete-inactive-destinations.html

        Note: that the index file(db.data) will not shrink, but should stabilize one the maximum is reached. The db.data file is a collection of data pages that can get reused once they are free, but unused destinations will need to be deleted if they are created at runtime.

        Show
        Gary Tully added a comment - To get the unused topics cleaned up, see: http://activemq.apache.org/delete-inactive-destinations.html Note: that the index file(db.data) will not shrink, but should stabilize one the maximum is reached. The db.data file is a collection of data pages that can get reused once they are free, but unused destinations will need to be deleted if they are created at runtime.
        Hide
        Krishnadasan T S added a comment -

        Thanks for your reply.

        I am going to change the run time creation logic of destination. Then will monitor the db.data size. thanks for your valuable suggestions.

        Show
        Krishnadasan T S added a comment - Thanks for your reply. I am going to change the run time creation logic of destination. Then will monitor the db.data size. thanks for your valuable suggestions.
        Hide
        Rob Davies added a comment -

        downgraded to major - let us know if this is now resolved in your environment

        Show
        Rob Davies added a comment - downgraded to major - let us know if this is now resolved in your environment
        Hide
        Timothy Bish added a comment -

        No feedback on resolution using newer version, assuming that its fixed.

        Show
        Timothy Bish added a comment - No feedback on resolution using newer version, assuming that its fixed.
        Hide
        Krishnadasan T S added a comment -

        As per my previous reply we have made changes to our application, It will go to production with in a month then only i can give the status. Thanks for your reply.

        Show
        Krishnadasan T S added a comment - As per my previous reply we have made changes to our application, It will go to production with in a month then only i can give the status. Thanks for your reply.

          People

          • Assignee:
            Unassigned
            Reporter:
            Krishnadasan T S
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development