Wicket
  1. Wicket
  2. WICKET-5147

WicketTester MockHttpRequest.getCookies very slow / OutOfMemory

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 6.7.0
    • Fix Version/s: 6.8.0
    • Component/s: None
    • Labels:
      None

      Description

      We have an extensive set of WicketTester tests. Recently, the wicket RELEASE in the maven repository changed to 6.7.0. After the new version, our tests got very slow.

      When profiling, I discovered that the MockHttpRequest.getCookies() was taking up a lot of time. Also, tests failed because of OutOfMemory exceptions. My guess is that somehow a lot of objects are created at such speeds that the GC cannot clean them

      I will investigate further, but switching back to 6.6.0 solved the issue.

      [Edit]
      The tests are run with TestNG and using 'mvn test'

      1. testcookies.zip
        28 kB
        Rob Audenaerde

        Issue Links

        1.
        cookies should expire Sub-task Resolved Unassigned
         

          Activity

          Hide
          Rob Audenaerde added a comment -

          During debugging a test, I see the number of Cookies is doubled a lot of times. This would explain the huge increase in memory.

          Show
          Rob Audenaerde added a comment - During debugging a test, I see the number of Cookies is doubled a lot of times. This would explain the huge increase in memory.
          Hide
          Rob Audenaerde added a comment -

          The Cookies all contain the same data.

          Show
          Rob Audenaerde added a comment - The Cookies all contain the same data.
          Hide
          Martin Grigorov added a comment -

          Please attach a test case that shows the problematic behavior.

          Show
          Martin Grigorov added a comment - Please attach a test case that shows the problematic behavior.
          Hide
          Rob Audenaerde added a comment -

          I will try to make one. It is a bit of work, as I need an AuthenticaedWebsession and (a lot of) ajax-calls

          Show
          Rob Audenaerde added a comment - I will try to make one. It is a bit of work, as I need an AuthenticaedWebsession and (a lot of) ajax-calls
          Hide
          Rob Audenaerde added a comment - - edited

          The test 'changeLabelTwoTimes' test will fail. The number of cookies in the request will be 16 after the first call (too much already) and is 64 after the second call.
          I would expect them to be 1.

          Show
          Rob Audenaerde added a comment - - edited The test 'changeLabelTwoTimes' test will fail. The number of cookies in the request will be 16 after the first call (too much already) and is 64 after the second call. I would expect them to be 1.
          Hide
          Michael Mosmann added a comment -

          think,i found it..

          Show
          Michael Mosmann added a comment - think,i found it..
          Hide
          Michael Mosmann added a comment -

          First Bugfix in https://github.com/apache/wicket/pull/35 ... but there should be a lot more tests.. so do NOT pull this, i will make an other refactoring with more test, hopefully without this reformating stuff..

          Show
          Michael Mosmann added a comment - First Bugfix in https://github.com/apache/wicket/pull/35 ... but there should be a lot more tests.. so do NOT pull this, i will make an other refactoring with more test, hopefully without this reformating stuff..
          Hide
          Michael Mosmann added a comment -

          I did a second try:

          https://github.com/apache/wicket/pull/36

          This way i fixed some bugs, wrote some tests, fixed the docu, fixed this reformating problem, and so on.

          Show
          Michael Mosmann added a comment - I did a second try: https://github.com/apache/wicket/pull/36 This way i fixed some bugs, wrote some tests, fixed the docu, fixed this reformating problem, and so on.
          Hide
          Martin Grigorov added a comment -

          Rob,

          Please check http://markmail.org/thread/an4g2mx5bpemc7nw
          Do you destroy WicketTester after your tests ?

          Show
          Martin Grigorov added a comment - Rob, Please check http://markmail.org/thread/an4g2mx5bpemc7nw Do you destroy WicketTester after your tests ?
          Hide
          Rob Audenaerde added a comment -

          I do not destroy the WicketTester. But I also do not reuse the WicketTester between tests. My UI tests use this piece of code:

          @BeforeMethod
          public final void setupTest() throws Exception

          { LOG.info( "Running test in class:" + this.getClass() ); this.wicketTester = new WicketTester( this.wicketApplication ); this.login(); }

          Besides that, I see the number of cookies increase expontentially in my testcase in the attached zip.

          Show
          Rob Audenaerde added a comment - I do not destroy the WicketTester. But I also do not reuse the WicketTester between tests. My UI tests use this piece of code: @BeforeMethod public final void setupTest() throws Exception { LOG.info( "Running test in class:" + this.getClass() ); this.wicketTester = new WicketTester( this.wicketApplication ); this.login(); } Besides that, I see the number of cookies increase expontentially in my testcase in the attached zip.
          Hide
          Martin Grigorov added a comment -

          We will fix the issue for Wicket 6.8.0. I'm just trying to tell you that you hit OOM because your tests leak resources.
          Cookie class is quite small. You have to produce a lot of instances to fill the memory. I doubt that you create so many cookies in a single test to run out of memory. It is just that resources from a previous usage of WicketTester are not recycled during the whole run of the test suite.

          I see you reuse the application instance for the different instances of WicketTester. This is already a candidate for memory leaks.
          I guess you do this to avoid costly initializations, like initializing Spring for example.

          Show
          Martin Grigorov added a comment - We will fix the issue for Wicket 6.8.0. I'm just trying to tell you that you hit OOM because your tests leak resources. Cookie class is quite small. You have to produce a lot of instances to fill the memory. I doubt that you create so many cookies in a single test to run out of memory. It is just that resources from a previous usage of WicketTester are not recycled during the whole run of the test suite. I see you reuse the application instance for the different instances of WicketTester. This is already a candidate for memory leaks. I guess you do this to avoid costly initializations, like initializing Spring for example.
          Hide
          Rob Audenaerde added a comment -

          Hi Martin,

          Thanks for the resolution. And you are right that I'm injecting the WicketAppliction using Spring

          BTW. The problem is not that the Cookie itself is big, but rather that going through some ajax calls within one test causes the number of cookies to double and double and double; so basically having one cookie will cause OOM after a slightly bigger test, (for example 20 ajaxcalls will cause at least 2^20 (~ 1 million) cookies.).

          Some context: In our application users can set filters in a few mouseclicks, but to make sure they have to do as little as possible, I want to remove all the options that are not applicable after each choice. Typically, the user will select 2 to 4 filters and some operators, so 'normal' behaviour would be around 4 to 8 ajax calls. I want to fuzztest this component. The number of filters and operations that I use in the fuzztest go up to 10, causing the OOM

          Show
          Rob Audenaerde added a comment - Hi Martin, Thanks for the resolution. And you are right that I'm injecting the WicketAppliction using Spring BTW. The problem is not that the Cookie itself is big, but rather that going through some ajax calls within one test causes the number of cookies to double and double and double; so basically having one cookie will cause OOM after a slightly bigger test, (for example 20 ajaxcalls will cause at least 2^20 (~ 1 million) cookies.). Some context: In our application users can set filters in a few mouseclicks, but to make sure they have to do as little as possible, I want to remove all the options that are not applicable after each choice. Typically, the user will select 2 to 4 filters and some operators, so 'normal' behaviour would be around 4 to 8 ajax calls. I want to fuzztest this component. The number of filters and operations that I use in the fuzztest go up to 10, causing the OOM
          Hide
          Michael Mosmann added a comment -

          Ok.

          • I have fixed it.
          • I have extended the test to check if cookies are visible in tester.getRequest().
          • I did check the quickstart.
          • I have updated comments
            Everything is fine.
          Show
          Michael Mosmann added a comment - Ok. I have fixed it. I have extended the test to check if cookies are visible in tester.getRequest(). I did check the quickstart. I have updated comments Everything is fine.
          Hide
          Michael Mosmann added a comment -

          Someone should have a look at https://github.com/apache/wicket/pull/36 for code review...

          Show
          Michael Mosmann added a comment - Someone should have a look at https://github.com/apache/wicket/pull/36 for code review...
          Hide
          Martin Grigorov added a comment -

          Thank you for the help, Michael!

          Show
          Martin Grigorov added a comment - Thank you for the help, Michael!

            People

            • Assignee:
              Martin Grigorov
              Reporter:
              Rob Audenaerde
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development