Geronimo
  1. Geronimo
  2. GERONIMO-3838

Close potential denial of service attack vector (OOM) in Tomcat session handling

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Critical Critical
    • Resolution: Fixed
    • Affects Version/s: 2.0.2, 2.1.1
    • Fix Version/s: 2.0.3, 2.1.4, 2.2
    • Component/s: Memory Leaks, security, Tomcat
    • Security Level: public (Regular issues)
    • Labels:
      None
    • Environment:

      tested with JDK 1.5 running on Windows XP and FreeBSD6.2

    • Patch Info:
      Patch Available

      Description

      There is memory leak and it can be repeated very easily, so it should be very easy to catch

      Install Geronimo and then run some kind of benchmarking software against its admin UI login page, for example
      program ab from Apache HTTP. This is realistic attack scenario, because lot of denial of service attacks are doing this (requesting one page many times).

      Watching memory used graph in admin console shows free memory slowly decreasing. After all available memory is exhausted, application server stops serving new requests and never restores ifself back to working state.

      I think that it is caused by allocating sessions without limiting total number of sessions to keep in memory and possibly to swap sessions out to file. There needs to be user-configurable setting for preventing this, it would be nice to add such setting to Admin console.

      Its very important to get this bug fixed.

        Activity

        Hide
        Radim Kolar added a comment -

        using Sun hat tool i can confirm that this problem is related to allocating lot of sessions.

        dump from HAT tool

        Instance Counts for All Classes (excluding platform)
        466128 instances of class [Ljava.util.concurrent.ConcurrentHashMap$HashEntry;
        83106 instances of class [C
        36330 instances of class [Ljava.lang.Object;
        30464 instances of class [Ljava.util.Hashtable$Entry;
        29133 instances of class [Ljava.util.concurrent.ConcurrentHashMap$Segment;
        29086 instances of class org.apache.catalina.session.StandardSession
        29086 instances of class org.apache.catalina.session.StandardSessionFacade

        There is difference between using IBM vs SUN JDK, If Sun jdk is used Geronimo never recovers from out of memory error.

        Proposed fix:
        Geronimo needs to swap inactive sessions to file or forget them and not to run out of memory. It would be nice to have number of sessions kept in main memory configurable via System configuration console.

        Show
        Radim Kolar added a comment - using Sun hat tool i can confirm that this problem is related to allocating lot of sessions. dump from HAT tool Instance Counts for All Classes (excluding platform) 466128 instances of class [Ljava.util.concurrent.ConcurrentHashMap$HashEntry; 83106 instances of class [C 36330 instances of class [Ljava.lang.Object; 30464 instances of class [Ljava.util.Hashtable$Entry; 29133 instances of class [Ljava.util.concurrent.ConcurrentHashMap$Segment; 29086 instances of class org.apache.catalina.session.StandardSession 29086 instances of class org.apache.catalina.session.StandardSessionFacade There is difference between using IBM vs SUN JDK, If Sun jdk is used Geronimo never recovers from out of memory error. Proposed fix: Geronimo needs to swap inactive sessions to file or forget them and not to run out of memory. It would be nice to have number of sessions kept in main memory configurable via System configuration console.
        Hide
        Rex Wang added a comment -

        it's still a problem in Geronimo 2.1.1

        Show
        Rex Wang added a comment - it's still a problem in Geronimo 2.1.1
        Hide
        Radim Kolar added a comment -

        Google search discovered that this issue can be avoided by switching to different Tomcat session manager, namely org.apache.catalina.session.PersistentManager.

        Show
        Radim Kolar added a comment - Google search discovered that this issue can be avoided by switching to different Tomcat session manager, namely org.apache.catalina.session.PersistentManager.
        Hide
        Donald Woods added a comment -

        affects multiple release streams

        Show
        Donald Woods added a comment - affects multiple release streams
        Hide
        Kevan Miller added a comment -

        Radim is correct. However, as best that I can tell, there's no way to configure a PersistentManager in Geronimo.

        https://issues.apache.org/jira/browse/GERONIMO-3376 would seem to have fixed this problem. However, I don't think it works.

        I looked at this a few weeks back. You can now generate a ManagerGBean, as described in GERONIMO-3376. However, there's nothing that actually hooks it into Tomcat configuration. So, Tomcat just creates it's own StandardManager (thus the GBean has no effect).

        Definitely something that needs to be fixed.

        Show
        Kevan Miller added a comment - Radim is correct. However, as best that I can tell, there's no way to configure a PersistentManager in Geronimo. https://issues.apache.org/jira/browse/GERONIMO-3376 would seem to have fixed this problem. However, I don't think it works. I looked at this a few weeks back. You can now generate a ManagerGBean, as described in GERONIMO-3376 . However, there's nothing that actually hooks it into Tomcat configuration. So, Tomcat just creates it's own StandardManager (thus the GBean has no effect). Definitely something that needs to be fixed.
        Hide
        Ivan added a comment - - edited

        I do some change in the ManagerGBean, so that it can could hander the initialization while set the className with org.apache.catalina.session.PersistentManager.
        For example :
        <manager>TomcatManager</manager>
        <gbean name="TomcatManager" class="org.apache.geronimo.tomcat.ManagerGBean">
        <attribute name="className">org.apache.catalina.session.PersistentManager</attribute>
        <attribute name="initParams">maxActiveSessions=10
        maxIdleBackup=10
        maxIdleSwap=11
        minIdleSwap=5
        store.className=org.apache.catalina.session.FileStore
        store.checkInterval=10
        store.directory=d:/testFolder/session</attribute>
        </gbean>
        We could set store.className=org.apache.catalina.session.JDBCStore to use the JDBCStore.

        One thing makes me confusion is that I found the reference name of manager and cluster in the TomcatWebAppContext.java is changed in the patch GERONIMO-3696, and it has conflict with the xml schema file http://geronimo.apache.org/xml/ns/j2ee/web/tomcat-2.0.1. I change those two reference name back, or it seems that the manager object could not be set. I am not sure whether I miss something, thanks very much for any comment !

        Show
        Ivan added a comment - - edited I do some change in the ManagerGBean, so that it can could hander the initialization while set the className with org.apache.catalina.session.PersistentManager. For example : <manager>TomcatManager</manager> <gbean name="TomcatManager" class="org.apache.geronimo.tomcat.ManagerGBean"> <attribute name="className">org.apache.catalina.session.PersistentManager</attribute> <attribute name="initParams">maxActiveSessions=10 maxIdleBackup=10 maxIdleSwap=11 minIdleSwap=5 store.className=org.apache.catalina.session.FileStore store.checkInterval=10 store.directory=d:/testFolder/session</attribute> </gbean> We could set store.className=org.apache.catalina.session.JDBCStore to use the JDBCStore. One thing makes me confusion is that I found the reference name of manager and cluster in the TomcatWebAppContext.java is changed in the patch GERONIMO-3696 , and it has conflict with the xml schema file http://geronimo.apache.org/xml/ns/j2ee/web/tomcat-2.0.1 . I change those two reference name back, or it seems that the manager object could not be set. I am not sure whether I miss something, thanks very much for any comment !
        Hide
        Ivan added a comment -

        I divided the patch for two files. The one Geronimo-3838-ManagerGBean.patch is to make ManagerGBean to support PersistenceManager configuration. And the other one is I am not quite sure. If I write <ManagerRetriever>TomcatManager</ManagerRetriever> in the plan file, it says that Manager element is expected, but if I write <Manager>TomcatManager</Manager>, it would say that no reference name "Manager" is found. Could anyone give some comments, for I found WADITomcatClusteringBuilder has reference to that static string. Thanks !

        Show
        Ivan added a comment - I divided the patch for two files. The one Geronimo-3838-ManagerGBean.patch is to make ManagerGBean to support PersistenceManager configuration. And the other one is I am not quite sure. If I write <ManagerRetriever>TomcatManager</ManagerRetriever> in the plan file, it says that Manager element is expected, but if I write <Manager>TomcatManager</Manager>, it would say that no reference name "Manager" is found. Could anyone give some comments, for I found WADITomcatClusteringBuilder has reference to that static string. Thanks !
        Hide
        Ivan added a comment -

        Just find the reason that mentioned in my previous comments, recreate the patch.
        1. Update the reference name of manager in the TomcatModuleBuilder to bring into correspondence with the name in the TomcatWebappContext.
        2. Update the ManageGBean to support the PersistentManager configuration.
        Please help to review the patch, thanks !

        Show
        Ivan added a comment - Just find the reason that mentioned in my previous comments, recreate the patch. 1. Update the reference name of manager in the TomcatModuleBuilder to bring into correspondence with the name in the TomcatWebappContext. 2. Update the ManageGBean to support the PersistentManager configuration. Please help to review the patch, thanks !
        Hide
        Donald Woods added a comment -

        Rev718945 in branches/2.1 (2.1.4-SNAPSHOT)

        Show
        Donald Woods added a comment - Rev718945 in branches/2.1 (2.1.4-SNAPSHOT)
        Hide
        Donald Woods added a comment -

        Rev718971 in branches/2.0 (2.0.3-SNAPSHOT)
        Rev718972 in trunk (2.2-SNAPSHOT)
        Leaving open until the automated builds complete and we have some TCK results from branches/2.1.

        Show
        Donald Woods added a comment - Rev718971 in branches/2.0 (2.0.3-SNAPSHOT) Rev718972 in trunk (2.2-SNAPSHOT) Leaving open until the automated builds complete and we have some TCK results from branches/2.1.
        Hide
        Donald Woods added a comment -

        updating title to properly reflect this as a potential DoS attack vector

        Show
        Donald Woods added a comment - updating title to properly reflect this as a potential DoS attack vector
        Hide
        Donald Woods added a comment -

        Automated build for 2.1.4 looked good and web related TCK buckets passed on 2.1.4.

        Show
        Donald Woods added a comment - Automated build for 2.1.4 looked good and web related TCK buckets passed on 2.1.4.
        Hide
        Ivan added a comment -

        For WADI is not used in Geronimo-2.0, so it seems that no need to update the reference name in the tomcatbuilder.
        I create a patch file for changing the name back.
        Thanks !

        Show
        Ivan added a comment - For WADI is not used in Geronimo-2.0, so it seems that no need to update the reference name in the tomcatbuilder. I create a patch file for changing the name back. Thanks !
        Hide
        Donald Woods added a comment -

        Updated patch from Ivan for the 2.0 branch applied as Rev719229

        Show
        Donald Woods added a comment - Updated patch from Ivan for the 2.0 branch applied as Rev719229

          People

          • Assignee:
            Donald Woods
            Reporter:
            Radim Kolar
          • Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development