Axis
  1. Axis
  2. AXIS-935

Using ThreadLocal Call objects in a servlet memory leak both on client and server side

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Resolution: Fixed
    • Affects Version/s: 1.1rc2
    • Fix Version/s: None
    • Component/s: Basic Architecture
    • Labels:
      None
    • Environment:
      Operating System: All
      Platform: All

      Description

      I discovered using Axis in tomcat causes a memory leak.

      It seems using threadlocal in the classes causes a memory leak for each Call
      that is created. Threadlocal sticks around until the calling thread dies, and
      since tomcat keeps threads alive to answer new calls, the threadlocal data
      never goes away. The Call object gets stuck in the threadlocal data, which
      contains the entire soap response and all objects. This is a BIG memory leak.
      I initially cleared it by forcing all Call objects to clean their member vars
      after I was done with a call, then I realized the source was from the thread
      local class. Here is some info I found about threadlocal:

      This is stated in the Sun Javadocs for ThreadLocal:

      Each thread holds an implicit reference to its copy of a
      ThreadLocal as long as the thread is alive and the
      ThreadLocal object is accessible; after a thread goes
      away, all of its copies of ThreadLocal variables are
      subject to garbage collection (unless other references
      to these copies exist).

      So, this means that ANY APPLICATION that uses PreparedStatements in a thread
      that 1) either does a lot of PreparedStatements or 2) never dies (i.e., a
      main thread) will ALWAYS eventually have an out of memory error. Simply put,
      this is a MEMORY LEAK. I imagine that the leak is very small, the
      ThreadLocal object only contains one member variable, maybe 64 bytes or less
      (depending on the VM implementation). So, our 60,000 PreparedStatements of 2
      ThreadLocals each times 64 bytes (my wild guess) is 7.5MB.

      Ideas? I've never used threadlocal myself so this is new to me.

        Issue Links

          Activity

          Jennifer Jackson created issue -
          Serge Knystautas made changes -
          Field Original Value New Value
          issue.field.bugzillaimportkey 21186 14876
          Hide
          Jennifer Jackson added a comment -

          Has anyone looked into this? I haven't used Axis because of this problem and doing a quick search it seems others have come across memory leaks also. I narrowed it down to the Call object not cleaning out its member vars (the whole Deserialization process). If you do that the only thing stuck in memory is the call object since its part of the ThreadLocal object.

          Show
          Jennifer Jackson added a comment - Has anyone looked into this? I haven't used Axis because of this problem and doing a quick search it seems others have come across memory leaks also. I narrowed it down to the Call object not cleaning out its member vars (the whole Deserialization process). If you do that the only thing stuck in memory is the call object since its part of the ThreadLocal object.
          Hide
          Davanum Srinivas added a comment -

          Do you have some sample code to recreate the problem?

          Show
          Davanum Srinivas added a comment - Do you have some sample code to recreate the problem?
          Davanum Srinivas made changes -
          Assignee Axis Developers Mailing List [ axis-dev@ws.apache.org ]
          Resolution Fixed [ 1 ]
          Status Open [ 1 ] Resolved [ 5 ]
          Hide
          Davanum Srinivas added a comment -

          sorry. closed by mistake. reopening...

          Show
          Davanum Srinivas added a comment - sorry. closed by mistake. reopening...
          Davanum Srinivas made changes -
          Resolution Fixed [ 1 ]
          Status Resolved [ 5 ] Reopened [ 4 ]
          Hide
          Davanum Srinivas added a comment -

          did you narrow down which ThreadLocal stuff is causing the problem? (which java class, approx line #'s?)

          thanks,
          dims

          Show
          Davanum Srinivas added a comment - did you narrow down which ThreadLocal stuff is causing the problem? (which java class, approx line #'s?) thanks, dims
          Hide
          Davanum Srinivas added a comment -

          Used Rational Purify and checked for Axis specific leaks and found none.

          – dims

          Show
          Davanum Srinivas added a comment - Used Rational Purify and checked for Axis specific leaks and found none. – dims
          Davanum Srinivas made changes -
          Status Reopened [ 4 ] Closed [ 6 ]
          Resolution Fixed [ 1 ]
          Hide
          Catalin Grigoroscuta added a comment -

          I confirm this bug in axis 1.1.
          It appears when you have axis in a web app, in WEB-INF/lib.
          Axis puts some some of its classes on thread locals (Tomcat's threads), which prevents the entire web app from being garbage collected when you stop it, because Tomcat's threads will keep strong reference to web app class loader through these thread locals.
          I will identify shortly these thread locals, and will try to correct them.
          Could anyone tell me how can I reopen this bug? Or do I have to create a new one?

          Show
          Catalin Grigoroscuta added a comment - I confirm this bug in axis 1.1. It appears when you have axis in a web app, in WEB-INF/lib. Axis puts some some of its classes on thread locals (Tomcat's threads), which prevents the entire web app from being garbage collected when you stop it, because Tomcat's threads will keep strong reference to web app class loader through these thread locals. I will identify shortly these thread locals, and will try to correct them. Could anyone tell me how can I reopen this bug? Or do I have to create a new one?
          Hide
          Guillaume Sauthier added a comment -

          Can you confirm this in Axis 1.2RC3 ?

          Show
          Guillaume Sauthier added a comment - Can you confirm this in Axis 1.2RC3 ?
          Hide
          Catalin Grigoroscuta added a comment -

          Actually, yes, the bug persists on 1.2RC3, althouhg I've seen there was some work done to fix it.
          I reproduce the bug with a WS server and a client running in the same web application (the client is actually a jsp).
          I've made a screenshot with the profiler screen that shows a lot of Call objects linked to tomcat's threads through ThreadLocalS (after stopping the web app, of course), but I do not know how to attach a file in JIRA (I'm a newbie to JIRA system).
          Also, I do't want to make public the name of the profiler on this list (it's a commercial one, I've used an evaluation version), as I don't want to be acused of making publicity.
          If you are interested, I can send you the screenshot (and profiler name) by email.

          Show
          Catalin Grigoroscuta added a comment - Actually, yes, the bug persists on 1.2RC3, althouhg I've seen there was some work done to fix it. I reproduce the bug with a WS server and a client running in the same web application (the client is actually a jsp). I've made a screenshot with the profiler screen that shows a lot of Call objects linked to tomcat's threads through ThreadLocalS (after stopping the web app, of course), but I do not know how to attach a file in JIRA (I'm a newbie to JIRA system). Also, I do't want to make public the name of the profiler on this list (it's a commercial one, I've used an evaluation version), as I don't want to be acused of making publicity. If you are interested, I can send you the screenshot (and profiler name) by email.
          Hide
          Steven Schwell added a comment -

          Yes, I confirm that 1.2RC3 is still leaking memory in a Thread-pooled server. Best guess is that it is related to use of ThreadLocals which are not released when a Thread is put back in the pool.

          Wish Java had a way to clear all ThreadLocals of a Thread. Failing that we need to provide a way to clear all ThreadLocals set by Axis. Alternatively, abandon use of ThreadLocals.

          This is high priority since it is a true memory leak. This bug should be reopened. Anyone know how to do that?

          Show
          Steven Schwell added a comment - Yes, I confirm that 1.2RC3 is still leaking memory in a Thread-pooled server. Best guess is that it is related to use of ThreadLocals which are not released when a Thread is put back in the pool. Wish Java had a way to clear all ThreadLocals of a Thread. Failing that we need to provide a way to clear all ThreadLocals set by Axis. Alternatively, abandon use of ThreadLocals. This is high priority since it is a true memory leak. This bug should be reopened. Anyone know how to do that?
          Hide
          Tom Jordahl added a comment -

          reopening per request

          Show
          Tom Jordahl added a comment - reopening per request
          Tom Jordahl made changes -
          Resolution Fixed [ 1 ]
          Status Closed [ 6 ] Reopened [ 4 ]
          Hide
          Tom Jordahl added a comment -

          The complex nature of storing the last Call object in the service was created to achieve thread safety for the service. That is, a service could be used by more than one thread. Since the Call object is stored in the service, the thread local data is to make sure that if thread T1 gets a stub from service S and thread T2 gets a stub from S, when both T1 and T2 each call an operation, T1 can call S.getLastCall() and get the call data for T1 operation, not T2.

          I hope that makes sense. It is not something that the current axis comitters designed or implemented ( the implementing parties have long 'left the building'). I hope my explanation helps clear up the design goals and why it works the way it works.

          If its any small consolation, you can probably be assured that Axis2 wont work this way (which is not to say that we shouldn't fix Axis 1.x).

          Show
          Tom Jordahl added a comment - The complex nature of storing the last Call object in the service was created to achieve thread safety for the service. That is, a service could be used by more than one thread. Since the Call object is stored in the service, the thread local data is to make sure that if thread T1 gets a stub from service S and thread T2 gets a stub from S, when both T1 and T2 each call an operation, T1 can call S.getLastCall() and get the call data for T1 operation, not T2. I hope that makes sense. It is not something that the current axis comitters designed or implemented ( the implementing parties have long 'left the building'). I hope my explanation helps clear up the design goals and why it works the way it works. If its any small consolation, you can probably be assured that Axis2 wont work this way (which is not to say that we shouldn't fix Axis 1.x).
          Hide
          Steven Schwell added a comment -

          I'm not convinced that the problem is with the ThreadLocal previousCall. It could be with any of the other uses of ThreadLocal in Axis. This is really a general Java problem with ThreadLocals and Thread pooling.

          If we cannot get away without using ThreadLocals then I think the best approach is to provide static method or methods to clear ALL ThreadLocals, similair to Service.clearCall().

          Show
          Steven Schwell added a comment - I'm not convinced that the problem is with the ThreadLocal previousCall. It could be with any of the other uses of ThreadLocal in Axis. This is really a general Java problem with ThreadLocals and Thread pooling. If we cannot get away without using ThreadLocals then I think the best approach is to provide static method or methods to clear ALL ThreadLocals, similair to Service.clearCall().
          Hide
          Catalin Grigoroscuta added a comment -

          You are right, this is a potential problem with all ThreadLocalS.
          Personally, I belive that relying strictly on ThreadLocalS is a bad programming technique (except for a few situation, like transaction context, etc), and when ThreadLocalS are used it must be done in try ... finally blocks (either explicit, or document the methods that create them, so that the caller knows how to use them correctly).
          For AXIS, I would be very happy with the proposed solution, as for the moment the only solution I quickly hacked is to record all thread locals as they are created by AXIS, and to clear them all when the app is shut down.
          Thought, doesn't this programming style remind you of C programming, where you must correctly pair the malloc/free calls?

          Show
          Catalin Grigoroscuta added a comment - You are right, this is a potential problem with all ThreadLocalS. Personally, I belive that relying strictly on ThreadLocalS is a bad programming technique (except for a few situation, like transaction context, etc), and when ThreadLocalS are used it must be done in try ... finally blocks (either explicit, or document the methods that create them, so that the caller knows how to use them correctly). For AXIS, I would be very happy with the proposed solution, as for the moment the only solution I quickly hacked is to record all thread locals as they are created by AXIS, and to clear them all when the app is shut down. Thought, doesn't this programming style remind you of C programming, where you must correctly pair the malloc/free calls?
          Hide
          Davanum Srinivas added a comment -

          removed threadlocals

          Show
          Davanum Srinivas added a comment - removed threadlocals
          Davanum Srinivas made changes -
          Status Reopened [ 4 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Mark Thomas made changes -
          Workflow jira [ 24871 ] Default workflow, editable Closed status [ 12555566 ]
          Mark Thomas made changes -
          Workflow Default workflow, editable Closed status [ 12555566 ] jira [ 12578917 ]
          Andreas Veithen made changes -
          Link This issue relates to AXIS-2850 [ AXIS-2850 ]

            People

            • Assignee:
              Unassigned
              Reporter:
              Jennifer Jackson
            • Votes:
              2 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development