Continuum
  1. Continuum
  2. CONTINUUM-1528

Continuum gets slower over time and eventually crashes

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.1-beta-3
    • Fix Version/s: 1.2
    • Component/s: None
    • Labels:
      None

      Description

      Our instance of continuum gets slower over time.
      We counted the time needed to verify all projects at night when nothing has changed in the SCM (hence, nothing is built).
      During day time, the usage was quite intensive, however.
      On day 1, it was about 3 minutes
      On day 2, it was about 7 minutes
      On day 3, it was about 10 minutes
      And after a few more days (perhaps a week or so), Continuum would constantly generate OutOfMemoryError and stop working.

      1. ci.png
        78 kB
        Julien S
      2. ci2.png
        81 kB
        Julien S
      3. ci3.png
        87 kB
        Julien S

        Activity

        Julien S created issue -
        Hide
        Emmanuel Venisse added a comment -

        How many projects in your instance?
        How many build results?
        build frequency?

        We have few instances that run since one month with more than 140 projects and we don't have OOME and it don't get slower over time.

        Show
        Emmanuel Venisse added a comment - How many projects in your instance? How many build results? build frequency? We have few instances that run since one month with more than 140 projects and we don't have OOME and it don't get slower over time.
        Hide
        Julien S added a comment -

        We have about 260 projects built every 15 minutes.

        I fully believe you do not run into the problem, but we have reproduced it 3 times already.

        Maybe could it be caused by the xmlrpc component?
        We are doing a pretty heavy usage of it (called every minute to check that every build is working).

        Show
        Julien S added a comment - We have about 260 projects built every 15 minutes. I fully believe you do not run into the problem, but we have reproduced it 3 times already. Maybe could it be caused by the xmlrpc component? We are doing a pretty heavy usage of it (called every minute to check that every build is working).
        Hide
        Emmanuel Venisse added a comment -

        I don't understand why you build all every 15 minutes. 260 projects checked every 15 minutes use lot of resources on your svn server and your continuum server.
        If servers can't accept all this charge, all actions are put in the java queue and can create OOME.

        Show
        Emmanuel Venisse added a comment - I don't understand why you build all every 15 minutes. 260 projects checked every 15 minutes use lot of resources on your svn server and your continuum server. If servers can't accept all this charge, all actions are put in the java queue and can create OOME.
        Hide
        Julien S added a comment -

        The two servers are alongside. The full check takes about 3 to 4 minutes when nothing is built. I agree that's a fairly heavy load. But the two servers seem to be able to cope with it. We do not do fresh builds of course !

        Usually during day time, in a 15m interval there are just a few commits. Continuum rebuilds them (and projects which depends on them) immediatly.

        You seem to suggest that the queue would fill up till exhaustion ? It is true that sometimes, the "next" schedule is triggered before the first one is completed, but at night (when there is no commit), Continuum seem to be able to process all the queue as it stops building and checking out projects (well, before the next schedule, I mean).

        Show
        Julien S added a comment - The two servers are alongside. The full check takes about 3 to 4 minutes when nothing is built. I agree that's a fairly heavy load. But the two servers seem to be able to cope with it. We do not do fresh builds of course ! Usually during day time, in a 15m interval there are just a few commits. Continuum rebuilds them (and projects which depends on them) immediatly. You seem to suggest that the queue would fill up till exhaustion ? It is true that sometimes, the "next" schedule is triggered before the first one is completed, but at night (when there is no commit), Continuum seem to be able to process all the queue as it stops building and checking out projects (well, before the next schedule, I mean).
        Hide
        Emmanuel Venisse added a comment -

        do you have lot of build results by project?

        Show
        Emmanuel Venisse added a comment - do you have lot of build results by project?
        Julien S made changes -
        Field Original Value New Value
        Attachment ci.png [ 30042 ]
        Hide
        Julien S added a comment -

        Most projects have one build result (but we also deploy sites), so we deploy one artifact and one site. A few projects have several artifacts.

        I have attached a image of the memory heap over time. I will attach an other one in a few days. There seem to be a upward trend, but it may be too early to confirm it.

        Show
        Julien S added a comment - Most projects have one build result (but we also deploy sites), so we deploy one artifact and one site. A few projects have several artifacts. I have attached a image of the memory heap over time. I will attach an other one in a few days. There seem to be a upward trend, but it may be too early to confirm it.
        Hide
        Emmanuel Venisse added a comment -

        Today, I removed some unused datas in the db that was loaded with build results. Can you test the latest snapshot to see if it fix your OOME issue?

        Show
        Emmanuel Venisse added a comment - Today, I removed some unused datas in the db that was loaded with build results. Can you test the latest snapshot to see if it fix your OOME issue?
        Hide
        Emmanuel Venisse added a comment -

        Do you have modified the maxmemory parameter in wrapper.conf. The default value is 384m. With 260 projects, I think you should increase it to 512m or 1024m.

        Show
        Emmanuel Venisse added a comment - Do you have modified the maxmemory parameter in wrapper.conf. The default value is 384m. With 260 projects, I think you should increase it to 512m or 1024m.
        Julien S made changes -
        Attachment ci2.png [ 30061 ]
        Hide
        Julien S added a comment -

        I have attached an other picture, which is confirming the likelihood of a "memory leak".
        Isn't possible that you keep in memory some data for every build result ?

        I will increase the maxmemory parameter in the wrapper as you suggested, but if there is indeed a "memory leaf", it will just save me a few days

        Otherwise, I'll do my best to try the latest snapshot, but, even with our heavy schedule, I will most likely need at least 4 days or so to test.
        I will keep you posted.

        Show
        Julien S added a comment - I have attached an other picture, which is confirming the likelihood of a "memory leak". Isn't possible that you keep in memory some data for every build result ? I will increase the maxmemory parameter in the wrapper as you suggested, but if there is indeed a "memory leaf", it will just save me a few days Otherwise, I'll do my best to try the latest snapshot, but, even with our heavy schedule, I will most likely need at least 4 days or so to test. I will keep you posted.
        Brett Porter made changes -
        Fix Version/s 1.2 [ 13779 ]
        Julien S made changes -
        Attachment ci3.png [ 31928 ]
        Hide
        Julien S added a comment -

        I have found some time to profile continuum 1.1.
        As seen in the third attachment, it would appear that the memory is still used up (albeit more slowly than previously).
        Unfortunately, it is very difficult to figure out what could cause this...

        Show
        Julien S added a comment - I have found some time to profile continuum 1.1. As seen in the third attachment, it would appear that the memory is still used up (albeit more slowly than previously). Unfortunately, it is very difficult to figure out what could cause this...
        Hide
        Emmanuel Venisse added a comment -

        I think the pb is with scm fileset and build output that go in the memory. Need to investigate on this.

        Show
        Emmanuel Venisse added a comment - I think the pb is with scm fileset and build output that go in the memory. Need to investigate on this.
        Hide
        Robin Roos added a comment -

        Hi Folks

        I thought I suffered from this problem. Then we added clean as one of the maven2 targets in our build definition. (Without clean we were seeing deleted classes remain in our build product.) Since adding clean I have seen stable performance from Continuum.

        I hope that info helps some people.

        Cheers, Robin.

        Show
        Robin Roos added a comment - Hi Folks I thought I suffered from this problem. Then we added clean as one of the maven2 targets in our build definition. (Without clean we were seeing deleted classes remain in our build product.) Since adding clean I have seen stable performance from Continuum. I hope that info helps some people. Cheers, Robin.
        Hide
        Marc Lustig added a comment -

        just to encircle the issue: we have deployed continuum on JBoss and we don't see such problems.

        Show
        Marc Lustig added a comment - just to encircle the issue: we have deployed continuum on JBoss and we don't see such problems.
        Hide
        Geert Pante added a comment -

        Some more probably irrelevant information:

        we have some 50 application servers running on SUN Java 1.5.0_07 for months with identical applications on almost identical hardware. The 51st we installed, crashed more or less every week with an OOME, with jconsole graphs similar to yours. We never found out where the memory leak was, but once we upgraded to SUN Java 1.6.0_02, it magically disappeared... FYI, the application was a JBossMQ 4.0.3sp1 with a small additional webapp on top.

        Show
        Geert Pante added a comment - Some more probably irrelevant information: we have some 50 application servers running on SUN Java 1.5.0_07 for months with identical applications on almost identical hardware. The 51st we installed, crashed more or less every week with an OOME, with jconsole graphs similar to yours. We never found out where the memory leak was, but once we upgraded to SUN Java 1.6.0_02, it magically disappeared... FYI, the application was a JBossMQ 4.0.3sp1 with a small additional webapp on top.
        Hide
        Emmanuel Venisse added a comment -

        Should be fixed by migration to spring and improvement in buid controller (CONTINUUM-1807)

        Show
        Emmanuel Venisse added a comment - Should be fixed by migration to spring and improvement in buid controller ( CONTINUUM-1807 )
        Emmanuel Venisse made changes -
        Resolution Fixed [ 1 ]
        Assignee Emmanuel Venisse [ evenisse ]
        Status Open [ 1 ] Closed [ 6 ]
        Mark Thomas made changes -
        Project Import Sun Apr 05 08:36:01 UTC 2015 [ 1428222961749 ]
        Mark Thomas made changes -
        Workflow jira [ 12710188 ] Default workflow, editable Closed status [ 12739848 ]
        Mark Thomas made changes -
        Project Import Sun Apr 05 21:12:18 UTC 2015 [ 1428268338676 ]
        Mark Thomas made changes -
        Workflow jira [ 12947384 ] Default workflow, editable Closed status [ 12985409 ]
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Closed Closed
        303d 20h 26m 1 Emmanuel Venisse 18/Aug/08 05:23

          People

          • Assignee:
            Emmanuel Venisse
            Reporter:
            Julien S
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development