Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Fixed
-
2.2.6
-
Windows XP / Solaris 10
JBoss 5.1.0.GA
-
Blocked on External
Description
I found that after undeploying my applications from JBoss 5.1.0.GA the classloaders remain hanging and are never garbage collected. These are the items I have tried so far:
- JVM options : -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
These make that the permgen gets collected periodically. - I have verified that after several redeployments there is indeed the PermGen OOM error.
The applications consist of a war package, containing the JSP frontend. and an EAR containing all the backend services (it includes CXF 2.2.6 and JAXB 2.1.12 within the EAR). I use CXF as a jaxws:client:
Analyzing a memory dump several minutes after undeployment, and after manually triggering GC, I extracted this with Eclipse MAT.
It contains the "Path to GC (Excluding all weak, soft and phantom references)":
Class Name | Shallow Heap | Retained Heap
---------------------------------------------------------------------------------------------------------------------------------------
org.jboss.classloader.spi.base.BaseClassLoader @ 0xdf9e538 | 96 | 10,484,208
'- <classloader> class com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty @ 0x2711fde0 | 8 | 8
|
40 | 1,336 | |
|
32 | 32 | |
| '- [39] java.lang.ThreadLocal$ThreadLocalMap$Entry[128] @ 0xf7ca2f8 | 528 | 23,240 | |
| '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xf6647a8 | 24 | 23,264 | |
| '- threadLocals java.lang.Thread @ 0xf4856c0 http-127.0.0.1-8080-14 Native Stack, Thread | 104 | 23,928 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
| '- Total: 7 entries | |||
|
40 | 1,280 | |
|
32 | 32 | |
| '- [9] java.lang.ThreadLocal$ThreadLocalMap$Entry[128] @ 0xf7bca90 | 528 | 22,520 | |
| '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xf515ca8 | 24 | 22,544 | |
| '- threadLocals java.lang.Thread @ 0xf35e078 http-127.0.0.1-8080-8 Native Stack, Thread | 104 | 23,208 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
|
32 | 32 | |
| '- Total: 11 entries |
'- Total: 2 entries | |
---------------------------------------------------------------------------------------------------------------------------------------
It shows the SingleElementLeafProperty from JAXBImpl holding references to the classloader (of the EAR packaged application). the instance SingleElementLeafProperty contains a fieldName which holds one of the parameter names of the invoked operation.
When I open a JPDA debug session to JBoss and suspend the Threads, I found indeed they included hard references to the SingleElementLeafProperty. This is a dump of the variables in the suspended Thread threadLocals:
[25] ThreadLocal$ThreadLocalMap$Entry (id=229)
discovered null
next ThreadLocal$ThreadLocalMap$Entry (id=229)
queue ReferenceQueue$Null (id=215)
referent null
value SingleElementLeafProperty<BeanT> (id=239)
acc AdaptedAccessor<BeanT,InMemValueT,OnWireValueT> (id=240)
defaultValue null
fieldName "msisdn" (id=244)
nillable false
propertyInfo null
tagName Name (id=245)
xacc TransducedAccessor$CompositeTransducedAccessorImpl<BeanT,ValueT> (id=247)
acc AdaptedAccessor<BeanT,InMemValueT,OnWireValueT> (id=240)
adapter Class<T> (javax.xml.bind.annotation.adapters.NormalizedStringAdapter) (id=253)
annotations null
annotationType null
cachedConstructor null
classRedefinedCount 0
declaredAnnotations null
declaredConstructors SoftReference<T> (id=349)
declaredFields null
declaredMethods null
declaredPublicFields null
declaredPublicMethods null
enumConstantDirectory null
enumConstants null
genericInfo ClassRepository (id=279)
lastRedefinedCount 0
name "javax.xml.bind.annotation.adapters.NormalizedStringAdapter" (id=283)
newInstanceCallerCache null
publicConstructors null
publicFields null
publicMethods null
core Accessor$FieldReflection<BeanT,ValueT> (id=254)
f Field (id=294)
annotations byte[20] (id=298)
clazz Class<T> (com.al.apc.generated.services.customeraccount.BaseAccountRequest) (id=300)
declaredAnnotations LinkedHashMap<K,V> (id=301)
fieldAccessor null
genericInfo null
modifiers 4
name "msisdn" (id=244)
override true
overrideFieldAccessor UnsafeObjectFieldAccessorImpl (id=303)
root Field (id=305)
securityCheckCache null
securityCheckTargetClassCache null
signature null
slot 7
type Class<T> (java.lang.String) (id=209)
valueType Class<T> (java.lang.String) (id=209)
staticAdapter null
valueType Class<T> (java.lang.String) (id=209)
xducer RuntimeBuiltinLeafInfoImpl$1 (id=260)
I have seen an old bug reference CXF-457 which clears the threadlocal at the end of invoke() method in JaxWsMethodInvoker. The proxy used in my case is JaxWsClientProxy. I found that after the method invocation the response is cleared from the threadlocals, but several parameter names remain there.
I think my setup is OK. We passed a lot of tests, including 3M successful generated (test)load transactions.
Any suggestions?
Kind Regards,
Danny