Cocoon
  1. Cocoon
  2. COCOON-2109

Incorrent cleanup of expired continuations

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 2.1.11
    • Fix Version/s: 2.1.12
    • Component/s: * Cocoon Core
    • Labels:
      None
    • Urgency:
      Normal
    • Other Info:
      Patch available
    • Affects version (Component):
      Cocoon Core
    • Fix version (Component):
      Cocoon Core

      Description

      The class ContinuationsManagerImpl is in charge of cleaning up expired continuations. It does so in the method expireContinuations. In this method there is a loop using an iterator over a SortedSet of continuations (WebContinuation). The loop is expecting that the continuations are ordered from oldest to newest. The loop stops in the first continuation that is not expired. The logic is correct since all the newer continuations could not be expired.

      However, the problem comes from the ordering of the continuations. To have the continuations ordered by lastAccessTime the program uses a TreeSet as a container of the continuations. The continuations implement the compareTo interface using the lastAccessTime and when a continuation is inserted in the container, it gets correctly ordered. But after the insertion, the continuation can change its lastAccessTime using the method WebContinuation.updateLastAccessTime() called from WebContinuation.getContinuation(). The ordering of the TreeSet is not updated with the change and when the program iterates over it, it does not get the continuations in the order expected.

      The result of this bug is that under hevy load many expired continuations may be around before the loop actually clean them up, eating memory resources and causing OutOfMemory.

      To fix it, a patch is provided that uses a HashSet for the continuations container and loops over all the continuations to check if they have expired.

        Activity

        Miguel Cuervo created issue -
        Miguel Cuervo made changes -
        Field Original Value New Value
        Description The class ContinuationsManagerImpl is in charge of cleaning up expired continuations. It does so in the method expireContinuations. In this method there is a loop using an iterator over a SortedSet of continuations (WebContinuation). The loop is expecting that the continuations are ordered from oldest to newest. The loop stops in the first continuation that is not expired. The logic is correct since all the newer continuations could not be expired.

        However, the problem comes from the ordering of the continuations. To have the continuations ordered by lastAccessTime the program uses a TreeSet as a container of the continuations. The continuations correctly implement the compareTo interface using the lastAccessTime and when a continuation is inserted in the container, it gets correctly ordered. But once the continuation is inserted in the container, the continuation can change its lastAccessTime using the method WebContinuation.updateLastAccessTime() called from WebContinuation.getContinuation(). The ordering of the TreeSet is not updated with the change and when the program iterates over it, it does not get the continuations in the order expected.

        The result of this bug is that under hevy load may expired continuations may be around before the loop actually clean them up, eating memory resources and causing OutOfMemory.

        To fix it, a patch is provided using a HashSet for the continuations container and looping over all the continuations to checked if they have expired.

        The class ContinuationsManagerImpl is in charge of cleaning up expired continuations. It does so in the method expireContinuations. In this method there is a loop using an iterator over a SortedSet of continuations (WebContinuation). The loop is expecting that the continuations are ordered from oldest to newest. The loop stops in the first continuation that is not expired. The logic is correct since all the newer continuations could not be expired.

        However, the problem comes from the ordering of the continuations. To have the continuations ordered by lastAccessTime the program uses a TreeSet as a container of the continuations. The continuations correctly implement the compareTo interface using the lastAccessTime and when a continuation is inserted in the container, it gets correctly ordered. But once the continuation is inserted in the container, the continuation can change its lastAccessTime using the method WebContinuation.updateLastAccessTime() called from WebContinuation.getContinuation(). The ordering of the TreeSet is not updated with the change and when the program iterates over it, it does not get the continuations in the order expected.

        The result of this bug is that under hevy load may expired continuations may be around before the loop actually clean them up, eating memory resources and causing OutOfMemory.

        To fix it, a patch is provided that uses a HashSet for the continuations container and loops over all the continuations to check if they have expired.

        Miguel Cuervo made changes -
        Attachment ContinuationsManagerImpl.java.patch [ 12363582 ]
        Miguel Cuervo made changes -
        Description The class ContinuationsManagerImpl is in charge of cleaning up expired continuations. It does so in the method expireContinuations. In this method there is a loop using an iterator over a SortedSet of continuations (WebContinuation). The loop is expecting that the continuations are ordered from oldest to newest. The loop stops in the first continuation that is not expired. The logic is correct since all the newer continuations could not be expired.

        However, the problem comes from the ordering of the continuations. To have the continuations ordered by lastAccessTime the program uses a TreeSet as a container of the continuations. The continuations correctly implement the compareTo interface using the lastAccessTime and when a continuation is inserted in the container, it gets correctly ordered. But once the continuation is inserted in the container, the continuation can change its lastAccessTime using the method WebContinuation.updateLastAccessTime() called from WebContinuation.getContinuation(). The ordering of the TreeSet is not updated with the change and when the program iterates over it, it does not get the continuations in the order expected.

        The result of this bug is that under hevy load may expired continuations may be around before the loop actually clean them up, eating memory resources and causing OutOfMemory.

        To fix it, a patch is provided that uses a HashSet for the continuations container and loops over all the continuations to check if they have expired.

        The class ContinuationsManagerImpl is in charge of cleaning up expired continuations. It does so in the method expireContinuations. In this method there is a loop using an iterator over a SortedSet of continuations (WebContinuation). The loop is expecting that the continuations are ordered from oldest to newest. The loop stops in the first continuation that is not expired. The logic is correct since all the newer continuations could not be expired.

        However, the problem comes from the ordering of the continuations. To have the continuations ordered by lastAccessTime the program uses a TreeSet as a container of the continuations. The continuations implement the compareTo interface using the lastAccessTime and when a continuation is inserted in the container, it gets correctly ordered. But after the insertion, the continuation can change its lastAccessTime using the method WebContinuation.updateLastAccessTime() called from WebContinuation.getContinuation(). The ordering of the TreeSet is not updated with the change and when the program iterates over it, it does not get the continuations in the order expected.

        The result of this bug is that under hevy load many expired continuations may be around before the loop actually clean them up, eating memory resources and causing OutOfMemory.

        To fix it, a patch is provided that uses a HashSet for the continuations container and loops over all the continuations to check if they have expired.

        Hide
        Vadim Gritsenko added a comment -
        It would be easier to fix tree set ordering, don't you think?
        Show
        Vadim Gritsenko added a comment - It would be easier to fix tree set ordering, don't you think?
        Hide
        Jörg Heinicke added a comment -
        Are you sure that modifying the value of the sort criteria does not change the order of the elements? At the end it's the question when the elements are sorted - on inserting them or on iterating over them. I'm pretty sure it's the latter and then there would be no problem. The Javadoc for SortedSet says: "A set that further guarantees that its iterator will traverse the set in ascending element order".

        A potential problem I can see is the modification of the lastAccessTime WHILE actually iterating over the set. The iterating itself is for sure synchronized to avoid ConcurrentModificationExceptions or something like this. But at that time you still can change the lastAccessTime. As soon as the ContinuationsManagerImpl gets to the first continuation with a modified lastAccessTime it would stop iterating and the expired continuations with a later lastAccessTime than the updated one had before would survive the cleanup.

        Does it make sense? (I did not look into the code.)
        Show
        Jörg Heinicke added a comment - Are you sure that modifying the value of the sort criteria does not change the order of the elements? At the end it's the question when the elements are sorted - on inserting them or on iterating over them. I'm pretty sure it's the latter and then there would be no problem. The Javadoc for SortedSet says: "A set that further guarantees that its iterator will traverse the set in ascending element order". A potential problem I can see is the modification of the lastAccessTime WHILE actually iterating over the set. The iterating itself is for sure synchronized to avoid ConcurrentModificationExceptions or something like this. But at that time you still can change the lastAccessTime. As soon as the ContinuationsManagerImpl gets to the first continuation with a modified lastAccessTime it would stop iterating and the expired continuations with a later lastAccessTime than the updated one had before would survive the cleanup. Does it make sense? (I did not look into the code.)
        Hide
        Miguel Cuervo added a comment - - edited
        It is not that easy. The update of the lastAccessTime it is done by other classes that use the reference to the continuation and that do not know anything about this set of continuations. Therefore you can not expected to have the continuation removed and inserted back in the set every time it changes to manteain the order.

        So the only solution I see it will be to recreate the sorted set every time that you want to iterate over it. And this operation will be probably slower that just iterate over all the continuations.
        Show
        Miguel Cuervo added a comment - - edited It is not that easy. The update of the lastAccessTime it is done by other classes that use the reference to the continuation and that do not know anything about this set of continuations. Therefore you can not expected to have the continuation removed and inserted back in the set every time it changes to manteain the order. So the only solution I see it will be to recreate the sorted set every time that you want to iterate over it. And this operation will be probably slower that just iterate over all the continuations.
        Hide
        Jörg Heinicke added a comment -
        Of course the other classes are not aware of this set - and should not since it is internal state of the ContinuationsManagerImpl. That's what I meant when I wrote you can not prevent updates on the lastAccessTime, only on the set itself.

        But you missed my point. Purely from the Javadoc I claim that the set is sorted on iterator() not on adding the continuations to the set. So there is no need to recreate the set on every iteration.
        Show
        Jörg Heinicke added a comment - Of course the other classes are not aware of this set - and should not since it is internal state of the ContinuationsManagerImpl. That's what I meant when I wrote you can not prevent updates on the lastAccessTime, only on the set itself. But you missed my point. Purely from the Javadoc I claim that the set is sorted on iterator() not on adding the continuations to the set. So there is no need to recreate the set on every iteration.
        Hide
        Vadim Gritsenko added a comment -
        Jorg - sort is happening on insert.

        Miguel - what I meant is to do
          set.remove(w);
          w.setLastAccessTime();
          set.add(w);

        Of course if somebody calls setLastAccessTime from outside, this code would have to be refactored if implementing approach above.
        Show
        Vadim Gritsenko added a comment - Jorg - sort is happening on insert. Miguel - what I meant is to do   set.remove(w);   w.setLastAccessTime();   set.add(w); Of course if somebody calls setLastAccessTime from outside, this code would have to be refactored if implementing approach above.
        Hide
        Vadim Gritsenko added a comment -
        PS Yes I agree that re-creating the set *is not* a solution. In this case iteration over complete set is acceptable.
        Show
        Vadim Gritsenko added a comment - PS Yes I agree that re-creating the set *is not* a solution. In this case iteration over complete set is acceptable.
        Hide
        Miguel Cuervo added a comment - - edited
        Jorg,

        Just a small test to prove that the ordering is happening on insert

        --------------------------

        package test;

        import java.util.Iterator;
        import java.util.SortedSet;
        import java.util.TreeSet;

        public class Test implements Comparable {

        public int i;

        public Test(int x) {
        i = x;
        }

        public int compareTo(Object o) {
        Test obj = (Test) o;
        return i - obj.i;
        }

        public static void main(String[] args) {
        SortedSet set = new TreeSet();

        Test obj1 = new Test(1);
        set.add(obj1);

        Test obj2 = new Test(3);
        set.add(obj2);

        Test obj3 = new Test(2);
        set.add(obj3);

        Iterator iter = set.iterator();
        System.out.println("Should print 1,2,3");
        while (iter.hasNext()) {
        Test obj = (Test) iter.next();
        System.out.println(obj.i);
        }



        obj3.i = 4;
        System.out.println("Order modified after insetion");
        System.out.println("Should print 1,3,4");
        iter = set.iterator();
        while (iter.hasNext()) {
        Test obj = (Test) iter.next();
        System.out.println(obj.i);
        }
        System.out.println("...but it prints 1,4,3 keeping the order in which the object was inserted");

        }
        }
        Show
        Miguel Cuervo added a comment - - edited Jorg, Just a small test to prove that the ordering is happening on insert -------------------------- package test; import java.util.Iterator; import java.util.SortedSet; import java.util.TreeSet; public class Test implements Comparable { public int i; public Test(int x) { i = x; } public int compareTo(Object o) { Test obj = (Test) o; return i - obj.i; } public static void main(String[] args) { SortedSet set = new TreeSet(); Test obj1 = new Test(1); set.add(obj1); Test obj2 = new Test(3); set.add(obj2); Test obj3 = new Test(2); set.add(obj3); Iterator iter = set.iterator(); System.out.println("Should print 1,2,3"); while (iter.hasNext()) { Test obj = (Test) iter.next(); System.out.println(obj.i); } obj3.i = 4; System.out.println("Order modified after insetion"); System.out.println("Should print 1,3,4"); iter = set.iterator(); while (iter.hasNext()) { Test obj = (Test) iter.next(); System.out.println(obj.i); } System.out.println("...but it prints 1,4,3 keeping the order in which the object was inserted"); } }
        Hide
        Jörg Heinicke added a comment -
        I committed a fix using Vadim's remove/add approach. The WebContinuation is only accessed via the ContinuationsManager, so it was easy to update the last access time whenever it is looked up. The update does no longer happen on WebContinuation.getContinuation() to not break the clean up. Assuming the WebContinuation is never hold across requests outside the ContinuationsManager this should have only the minimal downside of getting no update during request processing, but only at its very beginning. But I guess this is acceptable.
        Show
        Jörg Heinicke added a comment - I committed a fix using Vadim's remove/add approach. The WebContinuation is only accessed via the ContinuationsManager, so it was easy to update the last access time whenever it is looked up. The update does no longer happen on WebContinuation.getContinuation() to not break the clean up. Assuming the WebContinuation is never hold across requests outside the ContinuationsManager this should have only the minimal downside of getting no update during request processing, but only at its very beginning. But I guess this is acceptable.
        Jörg Heinicke made changes -
        Affects Version/s 2.1.9 [ 12310650 ]
        Assignee Jörg Heinicke [ joerg.heinicke@gmx.de ]
        Affects version (Component) Parent values: Cocoon Core(10151).
        Affects Version/s 2.1.10 [ 12310931 ]
        Component/s - Flowscript [ 12310444 ]
        Fix Version/s 2.1.12-dev (Current SVN) [ 12312903 ]
        Component/s * Cocoon Core [ 12310442 ]
        Fix version (Component) Parent values: Cocoon Core(10227).
        Jörg Heinicke made changes -
        Affects Version/s 2.1.6 [ 12310613 ]
        Affects Version/s 2.1.8 [ 12310601 ]
        Affects Version/s 2.1.7 [ 12310614 ]
        Jörg Heinicke made changes -
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]
        Hide
        Jörg Heinicke added a comment -
        The threading issues you addressed in your patch have not been addressed in my commit yet. It is not as easy as synchronizing on the sets before iterating over them because they have to be in sync.
        Show
        Jörg Heinicke added a comment - The threading issues you addressed in your patch have not been addressed in my commit yet. It is not as easy as synchronizing on the sets before iterating over them because they have to be in sync.
        Hide
        Jörg Heinicke added a comment -
        The threading issues in ContinuationsManagerImpl were fixed in rev http://svn.apache.org/viewvc?view=rev&revision=643295 (with few changes from 643294 necessary (WebContinuation is now Cloneable)).
        Show
        Jörg Heinicke added a comment - The threading issues in ContinuationsManagerImpl were fixed in rev http://svn.apache.org/viewvc?view=rev&revision=643295 (with few changes from 643294 necessary (WebContinuation is now Cloneable)).

          People

          • Assignee:
            Jörg Heinicke
            Reporter:
            Miguel Cuervo
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development