Multiple deploy/undeploy leaks memory in PCRegistry

• Key: OPENJPA-285
• Type: Bug
• Status: Resolved
• Resolution: Fixed
• Priority: Major
• Assignee: Kevin Sutter
• Reporter: Pinaki Poddar
• Component/s: kernel
• Affects Version/s: 1.0.0
• Fix Version/s: 1.0.0

Description

Kevin Miller reported:
Geronimo is running out of PermGen space in some simple deploy/ undeploy scenarios involving OpenJPA. The cause of the problem seems to be the _metas table in PCRegistry. _metas is a ConcurrentReferenceHashMap with WEAK reference keys and HARD reference values. The keys are the PersistenceCapable classes. While the values are the metadata for these classes which are maintained by the internal Meta class.

The cause of the ClassLoader memory leak is simple -- if any of the objects/classes held by the Meta class (e.g. fieldTypes) have also been loaded by the same ClassLoader used to load the PersistenceCapable class, the PersistenceCapable class (the weak key) will never be GCed. The value of the HashMap entry will always maintain a hard reference to the ClassLoader. Since the ClassLoader will never be GC'ed, the the the pcClass Class object will never be GC'able...

The problem can be easily recreated using current Geronimo trunk and the Geronimo Daytrader application.

Patrick Linskey suggested:
Change PCRegistry.fieldTypes to be String[] instead of Class[], and dematerialize them as needed.

Robert Burrell Donkin/Marc Prud'hommeaux both pointed out that alternatives such as to
listen for the death of a ClassLoader and manually unregistering metadata would be more costly in terms of complexity.

This patch follows Patrick's suggestion.
1. Changes the Meta.fieldTypes to String[] from Class[]
2. Adapts the enhanced bytecode accordingly to the modified method signatures
3. PCRegistry getFieldTypes() load the fields' declared type using the same loader that loaded the owner pc class.

Note: For a class C and its field f, CL(c) == CL(f) is not always true. (Kevin Miller)
          But CL(c) will be able to load declared type of f either directly or via one of its parent (Craig Russel)

Comments

[Pinaki Poddar]

This patch
1. Changes Meta.fieldTypes from Class[] to String[]
2. Adjusts PCEnhancer to the modified method signature
3. Loads each field's type in PCRegistry.getFieldTypes() using the classloader of pc class
 
Tested for regression against OpenJPA testcases.
Performance hit is a concern.

[Pinaki Poddar]

Would it be possible to run this patch in Geronimo environment?

This patch
1. changes all field types of Class to String in PCRegistry$Meta
2. Retains all method signature the same. The constructor of Meta translates all input Class arguments to their stringfied form
3. Reverts all changes of the earlier patch in PCEnhancer
4. The class names held in PCRegistry$Meta are converted to Classes in PCRegistry. The classloader used to load these classes is always the same classloader that loaded the user-defined persistence capable class.

[Kevan Miller]

I'm attaching a potential fix for this problem. It adds a PCRegistry.deRegister(ClassLoader) method. It allows Geronimo to inform OpenJPA when a ClassLoader is no longer valid. deRegister() simply iterates through all entries in the _metas map and removes all entries whose keys were loaded by the associated ClassLoader. If you don't like iterating though all of the _metas tables, it's a simple matter to maintain a set of Classes associated with each ClassLoader. I like the simplicity of it iterating. ClassLoaders are removed on an infrequent basis.

I've tested this change along with associated Geronimo changes in https://issues.apache.org/jira/browse/GERONIMO-3326. I've been through 100's of deploy/undeploy cycles without a problem...

The problem with other solutions (WeakReferences or Stringified Class names) is that PCRegistry$Meta.pc must be a hard reference. It is your only reference to the PersistenceCapable object. If it is a WeakReference, the PersistenceCapable object will be GC'ed and bad things start to happen... ;-)

There's one other solution, which could be considered: Allow embedders to be notified when you've created PersistenceCapable objects. Embedders could maintain the strong references to these objects and delete the references when a ClassLoader has been deleted. PCRegistry references could then be WeakReferences.

[Kevan Miller]

Found another ClassLoader memory leak caused by ImplHelper. This one is pretty straight forward.

Will be doing a bit more testing...

[Donald Woods]

Is there any chance we can get this into the 1.0.0 stream before 8/3, which is when we will start the final TCK runs for Geronimo 2.0?

[Kevin Sutter]

Don and Kevan,
I just went ahead with the integration of Kevan's patches into OpenJPA. Per the e-mail conversation that we had about these changes, I am still a little hesitant with the new PCRegistry.deRegister() method. But, as you have pointed out, there haven't been any other solutions suggested that resolve the problem. Although this solution puts the responsibility on the embedder of OpenJPA, it does provide the necessary communication between the Embedder and OpenJPA that this particular class loader is no longer required.

I have no concerns with Kevan's other patch for the ImplHelper to use a ConcurrentReferenceHashMap.

Thanks for digging into these problems and providing these patches.

Kevin

[Kevin Sutter]

Resolved per the discussion on the dev mailing list (http://www.nabble.com/PCRegistry-ClassLoader-memory-leak-tf4091308.html) and this Issue.

[Kevin Sutter]

Also, the two other patches attached to this Issue (JIRA-285.patch.txt and JIRA-285.patch.2.txt) are not being included in this commit processing. Neither of these were deemed to resolve the problem as originally reported.