Cayenne
  1. Cayenne
  2. CAY-770

bug / memory leak in DispatchQueue and EventManager

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2 branch
    • Fix Version/s: 3.0M2
    • Component/s: Core Library
    • Labels:
      None
    • Environment:
      Linux Kernel: 2.16.2
      OS: Debian
      Tomcat: 5.5.16
      Axis: 1.3
      Java: Standart Edition 1.5.0_10
      Java Args:
        -Xms512M
        -Xmx1024M

      Description

      First a little bit about my setup:

      I have 3 servers. Each server runs an axis service. The service uses cayenne 1.2.1 to connect to a database. It reads customer and account information from the DB etc..

      The servers are using cayenne's shared caching with javagroups as the messaging service so that changes made from one server are dispatched to the other servers.

      The avarage connections per second is somewhere around 4-5 for each server.

      However with nearly constant and unchanging usage the load of each server increases over time.
      To further test this I created a test server with a similar setup but deactivated the shared caching and JavaGroups setup. There I created a test program that creates totally constant usage of exactly 4 request per second.
      Still the load of the server is increasing until the cpu load is so high that the requests can not be processed anymore.

      I installed a java profiler to trying to pinpoint the location of this error and this is what I found out.

      I let the server run for 24 hours and then stopped the program which creates the test usage.

      But even while the server was idle there where still a lot of instances in the java heap. I even manually started the GC a few times.

      java.util.HashMap$Entry (almost 1.5 milion insances)
      java.lang.ref.WeakReference (over 1 million insances)
      org.objectstyle.cayenne.util.Invocation (over 1 million insances)
      org.objectstyle.cayenne.access.DataContext (only 30 Instances)

      When the LoadTest is running 58% of the cpu time is used within HashSet.add() which is called within

      org.objectstyle.cayenne.event.DispatchQueue.addInvocation

      The longer the test is running the higher the percentage of cpu time used from addInvocation. The problem seems to be that there are some hard references to the Invocations objects so that the WeakReferences are not destroyed and the HashSet gets bigger and bigger.

      1. memta6.jpg
        236 kB
        Ayhan Kondoz
      2. memorysy5.jpg
        46 kB
        Ayhan Kondoz
      3. cpuci0.jpg
        139 kB
        Ayhan Kondoz
      4. CAY-770.patch
        0.9 kB
        Phil Larson

        Activity

        Hide
        Ayhan Kondoz added a comment -

        Instances of Invocations

        Show
        Ayhan Kondoz added a comment - Instances of Invocations
        Hide
        Ayhan Kondoz added a comment -

        Shows the number of DataContext Instances

        Show
        Ayhan Kondoz added a comment - Shows the number of DataContext Instances
        Hide
        Ayhan Kondoz added a comment -

        cpu profiling

        Show
        Ayhan Kondoz added a comment - cpu profiling
        Hide
        Phil Larson added a comment -

        Patched against 3.0M1 tag

        Show
        Phil Larson added a comment - Patched against 3.0M1 tag
        Hide
        Phil Larson added a comment -

        I was having the same issue with a project. After some inspection with a debugger and memory profiler I've come to the attached patch.

        Essentially, expired Invocations will not be removed until the event is misfired. In our project, those events are never attempted to be fired, but the listeners are added everytime we create a new DataContext. So they pile up indefinitely and eventually we run out of memory.

        This patch modifies the add method to check for Invocations with null targets. Too bad there isn't a WeakHashSet.

        Show
        Phil Larson added a comment - I was having the same issue with a project. After some inspection with a debugger and memory profiler I've come to the attached patch. Essentially, expired Invocations will not be removed until the event is misfired. In our project, those events are never attempted to be fired, but the listeners are added everytime we create a new DataContext. So they pile up indefinitely and eventually we run out of memory. This patch modifies the add method to check for Invocations with null targets. Too bad there isn't a WeakHashSet.
        Hide
        Andrus Adamchik added a comment -

        Phil, thanks! I think you are absolutely right about the cause and thanks for the patch. Let me test this a little more and commit the fix.

        Show
        Andrus Adamchik added a comment - Phil, thanks! I think you are absolutely right about the cause and thanks for the patch. Let me test this a little more and commit the fix.
        Hide
        Andrus Adamchik added a comment -

        applied to all three branches with minor modifications. Appreciate if Ayhan and Phil could confirm on whether this fixed the issue , then we can close the bug report.

        Thanks
        Andrus

        Show
        Andrus Adamchik added a comment - applied to all three branches with minor modifications. Appreciate if Ayhan and Phil could confirm on whether this fixed the issue , then we can close the bug report. Thanks Andrus
        Hide
        Phil Larson added a comment -

        That works for me. Although you have a typo in your comment: "as failure to do taht can".

        Show
        Phil Larson added a comment - That works for me. Although you have a typo in your comment: "as failure to do taht can".

          People

          • Assignee:
            Andrus Adamchik
            Reporter:
            Ayhan Kondoz
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development