[CAY-770] bug / memory leak in DispatchQueue and EventManager

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Fixed
• Affects Version/s: 1.2 branch
• Fix Version/s: 3.0M2
• Component/s: Core Library
• Environment: Linux Kernel: 2.16.2; OS: Debian; Tomcat: 5.5.16; Axis: 1.3; Java: Standart Edition 1.5.0_10; Java Args:; -Xms512M; -Xmx1024M
• Assignee: Andrus Adamchik
• Reporter: Ayhan Kondoz
• Created: 15/Mar/07 17:18
• Updated: 09/Nov/08 01:07
• Resolved: 22/Sep/07 10:19

Description

First a little bit about my setup:

I have 3 servers. Each server runs an axis service. The service uses cayenne 1.2.1 to connect to a database. It reads customer and account information from the DB etc..

The servers are using cayenne's shared caching with javagroups as the messaging service so that changes made from one server are dispatched to the other servers.

The avarage connections per second is somewhere around 4-5 for each server.

However with nearly constant and unchanging usage the load of each server increases over time.
To further test this I created a test server with a similar setup but deactivated the shared caching and JavaGroups setup. There I created a test program that creates totally constant usage of exactly 4 request per second.
Still the load of the server is increasing until the cpu load is so high that the requests can not be processed anymore.

I installed a java profiler to trying to pinpoint the location of this error and this is what I found out.

I let the server run for 24 hours and then stopped the program which creates the test usage.

But even while the server was idle there where still a lot of instances in the java heap. I even manually started the GC a few times.

java.util.HashMap$Entry (almost 1.5 milion insances)
java.lang.ref.WeakReference (over 1 million insances)
org.objectstyle.cayenne.util.Invocation (over 1 million insances)
org.objectstyle.cayenne.access.DataContext (only 30 Instances)

When the LoadTest is running 58% of the cpu time is used within HashSet.add() which is called within

org.objectstyle.cayenne.event.DispatchQueue.addInvocation

The longer the test is running the higher the percentage of cpu time used from addInvocation. The problem seems to be that there are some hard references to the Invocations objects so that the WeakReferences are not destroyed and the HashSet gets bigger and bigger.