Cayenne
  1. Cayenne
  2. CAY-770

bug / memory leak in DispatchQueue and EventManager

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 1.2 branch
    • Fix Version/s: 3.0M2
    • Component/s: Core Library
    • Labels:
      None
    • Environment:
      Linux Kernel: 2.16.2
      OS: Debian
      Tomcat: 5.5.16
      Axis: 1.3
      Java: Standart Edition 1.5.0_10
      Java Args:
        -Xms512M
        -Xmx1024M

      Description

      First a little bit about my setup:

      I have 3 servers. Each server runs an axis service. The service uses cayenne 1.2.1 to connect to a database. It reads customer and account information from the DB etc..

      The servers are using cayenne's shared caching with javagroups as the messaging service so that changes made from one server are dispatched to the other servers.

      The avarage connections per second is somewhere around 4-5 for each server.

      However with nearly constant and unchanging usage the load of each server increases over time.
      To further test this I created a test server with a similar setup but deactivated the shared caching and JavaGroups setup. There I created a test program that creates totally constant usage of exactly 4 request per second.
      Still the load of the server is increasing until the cpu load is so high that the requests can not be processed anymore.

      I installed a java profiler to trying to pinpoint the location of this error and this is what I found out.

      I let the server run for 24 hours and then stopped the program which creates the test usage.

      But even while the server was idle there where still a lot of instances in the java heap. I even manually started the GC a few times.

      java.util.HashMap$Entry (almost 1.5 milion insances)
      java.lang.ref.WeakReference (over 1 million insances)
      org.objectstyle.cayenne.util.Invocation (over 1 million insances)
      org.objectstyle.cayenne.access.DataContext (only 30 Instances)

      When the LoadTest is running 58% of the cpu time is used within HashSet.add() which is called within

      org.objectstyle.cayenne.event.DispatchQueue.addInvocation

      The longer the test is running the higher the percentage of cpu time used from addInvocation. The problem seems to be that there are some hard references to the Invocations objects so that the WeakReferences are not destroyed and the HashSet gets bigger and bigger.

      1. CAY-770.patch
        0.9 kB
        Phil Larson
      2. cpuci0.jpg
        139 kB
        Ayhan Kondoz
      3. memta6.jpg
        236 kB
        Ayhan Kondoz
      4. memorysy5.jpg
        46 kB
        Ayhan Kondoz

        Activity

        Ari Maniatis made changes -
        Workflow jira [ 12456634 ] Cayenne workflow [ 12487439 ]
        Henri Yandell made changes -
        Project Import Sat Mar 21 00:51:04 PDT 2009 [ 1237621864637 ]
        Ari Maniatis made changes -
        Fix Version/s 3.0M2 [ 10123 ]
        Fix Version/s 2.0 [STABLE] [ 10092 ]
        Fix Version/s 1.2 [STABLE] [ 10030 ]
        Fix Version/s 3.0 [ 10091 ]
        Andrus Adamchik made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Fix Version/s 3.0 [ 10091 ]
        Fix Version/s 1.2 [STABLE] [ 10030 ]
        Fix Version/s 2.0 [STABLE] [ 10092 ]
        Resolution Fixed [ 1 ]
        Phil Larson made changes -
        Attachment CAY-770.patch [ 10591 ]
        Ayhan Kondoz made changes -
        Attachment cpuci0.jpg [ 10563 ]
        Ayhan Kondoz made changes -
        Attachment memta6.jpg [ 10562 ]
        Ayhan Kondoz made changes -
        Field Original Value New Value
        Attachment memorysy5.jpg [ 10561 ]
        Ayhan Kondoz created issue -

          People

          • Assignee:
            Andrus Adamchik
            Reporter:
            Ayhan Kondoz
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development