Uploaded image for project: 'CXF'
  1. CXF
  2. CXF-2689

Classloader not GC'ed after undeploying the applications. Reference from SingleElementLeafProperty

    XMLWordPrintableJSON

    Details

    • CXF Fields:
      Blocked on External

      Description

      I found that after undeploying my applications from JBoss 5.1.0.GA the classloaders remain hanging and are never garbage collected. These are the items I have tried so far:

      • JVM options : -XX:+UseConcMarkSweepGC -XX:+CMSPermGenSweepingEnabled -XX:+CMSClassUnloadingEnabled
        These make that the permgen gets collected periodically.
      • I have verified that after several redeployments there is indeed the PermGen OOM error.

      The applications consist of a war package, containing the JSP frontend. and an EAR containing all the backend services (it includes CXF 2.2.6 and JAXB 2.1.12 within the EAR). I use CXF as a jaxws:client:

      Analyzing a memory dump several minutes after undeployment, and after manually triggering GC, I extracted this with Eclipse MAT.
      It contains the "Path to GC (Excluding all weak, soft and phantom references)":

      Class Name | Shallow Heap | Retained Heap
      ---------------------------------------------------------------------------------------------------------------------------------------
      org.jboss.classloader.spi.base.BaseClassLoader @ 0xdf9e538 | 96 | 10,484,208
      '- <classloader> class com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty @ 0x2711fde0 | 8 | 8

      • <class> com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty @ 0xeeb0140
      40 1,336
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x10173b88
      32 32
        '- [39] java.lang.ThreadLocal$ThreadLocalMap$Entry[128] @ 0xf7ca2f8 528 23,240
        '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xf6647a8 24 23,264
        '- threadLocals java.lang.Thread @ 0xf4856c0 http-127.0.0.1-8080-14 Native Stack, Thread 104 23,928
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x101d8f98
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x101d9240
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x101e8df0
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x103c01f8
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x104c1b20
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x10567780
      32 32
      '- Total: 7 entries  
      • <class> com.sun.xml.bind.v2.runtime.property.SingleElementLeafProperty @ 0xed70788
      40 1,280
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xf8dcc78
      32 32
        '- [9] java.lang.ThreadLocal$ThreadLocalMap$Entry[128] @ 0xf7bca90 528 22,520
        '- table java.lang.ThreadLocal$ThreadLocalMap @ 0xf515ca8 24 22,544
        '- threadLocals java.lang.Thread @ 0xf35e078 http-127.0.0.1-8080-8 Native Stack, Thread 104 23,208
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xf8dcd18
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xf8e7628
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd624d0
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd6d480
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd77710
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd96740
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xfd9d5c0
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0xff51780
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x10145a28
      32 32
       
      • value java.lang.ThreadLocal$ThreadLocalMap$Entry @ 0x106537c8
      32 32
      '- Total: 11 entries  

      '- Total: 2 entries | |
      ---------------------------------------------------------------------------------------------------------------------------------------

      It shows the SingleElementLeafProperty from JAXBImpl holding references to the classloader (of the EAR packaged application). the instance SingleElementLeafProperty contains a fieldName which holds one of the parameter names of the invoked operation.

      When I open a JPDA debug session to JBoss and suspend the Threads, I found indeed they included hard references to the SingleElementLeafProperty. This is a dump of the variables in the suspended Thread threadLocals:
      [25] ThreadLocal$ThreadLocalMap$Entry (id=229)
      discovered null
      next ThreadLocal$ThreadLocalMap$Entry (id=229)
      queue ReferenceQueue$Null (id=215)
      referent null
      value SingleElementLeafProperty<BeanT> (id=239)
      acc AdaptedAccessor<BeanT,InMemValueT,OnWireValueT> (id=240)
      defaultValue null
      fieldName "msisdn" (id=244)
      nillable false
      propertyInfo null
      tagName Name (id=245)
      xacc TransducedAccessor$CompositeTransducedAccessorImpl<BeanT,ValueT> (id=247)
      acc AdaptedAccessor<BeanT,InMemValueT,OnWireValueT> (id=240)
      adapter Class<T> (javax.xml.bind.annotation.adapters.NormalizedStringAdapter) (id=253)
      annotations null
      annotationType null
      cachedConstructor null
      classRedefinedCount 0
      declaredAnnotations null
      declaredConstructors SoftReference<T> (id=349)
      declaredFields null
      declaredMethods null
      declaredPublicFields null
      declaredPublicMethods null
      enumConstantDirectory null
      enumConstants null
      genericInfo ClassRepository (id=279)
      lastRedefinedCount 0
      name "javax.xml.bind.annotation.adapters.NormalizedStringAdapter" (id=283)
      newInstanceCallerCache null
      publicConstructors null
      publicFields null
      publicMethods null
      core Accessor$FieldReflection<BeanT,ValueT> (id=254)
      f Field (id=294)
      annotations byte[20] (id=298)
      clazz Class<T> (com.al.apc.generated.services.customeraccount.BaseAccountRequest) (id=300)
      declaredAnnotations LinkedHashMap<K,V> (id=301)
      fieldAccessor null
      genericInfo null
      modifiers 4
      name "msisdn" (id=244)
      override true
      overrideFieldAccessor UnsafeObjectFieldAccessorImpl (id=303)
      root Field (id=305)
      securityCheckCache null
      securityCheckTargetClassCache null
      signature null
      slot 7
      type Class<T> (java.lang.String) (id=209)
      valueType Class<T> (java.lang.String) (id=209)
      staticAdapter null
      valueType Class<T> (java.lang.String) (id=209)
      xducer RuntimeBuiltinLeafInfoImpl$1 (id=260)

      I have seen an old bug reference CXF-457 which clears the threadlocal at the end of invoke() method in JaxWsMethodInvoker. The proxy used in my case is JaxWsClientProxy. I found that after the method invocation the response is cleared from the threadlocals, but several parameter names remain there.

      I think my setup is OK. We passed a lot of tests, including 3M successful generated (test)load transactions.

      Any suggestions?

      Kind Regards,
      Danny

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              dannyblazej Danny Blazejczak
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: