Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1.3
    • Fix Version/s: 1.1.4
    • Component/s: Context and Scopes
    • Environment:
      Tomcat 7.0.22, OWB 1.1.3, memcached-session-manager 1.6.1
      Should be unrelated: MyFaces 2.1.6, CODI 1.0.3, ExtVal 2.0.5, Hibernate 3.6.9, BVAL 0.3

      Description

      Martin (from msm) and I created an test app for this issue last year: https://github.com/magro/msm-sample-webapp/tree/openwebbeans and this was fixed within OWB-523.
      However, if i upgrade the Tomcat to 7.0.22 or newer, the bean replication does not work again.
      As you can see in this test app, the normal failover works (right input field) but the bean failover (left input field) does not work.

      I tested several Tomcat version and 7.0.21 or older works without problems, so the problem occured with Tomat 7.0.22.

      I don't know exactly if the problem is related to the OWB failover mechanism or Tomcat

      1. OWB-646.patch
        2 kB
        Mark Struberg

        Issue Links

          Activity

          Hide
          Mark Struberg added a comment -

          Thomas, can you please retry this with the lastest OWB 1.1.4-SNAPSHOT ?
          In the meantime I'll create a unit test for the FailOverService.

          Show
          Mark Struberg added a comment - Thomas, can you please retry this with the lastest OWB 1.1.4-SNAPSHOT ? In the meantime I'll create a unit test for the FailOverService.
          Hide
          Thomas Andraschko added a comment -

          Hi Mark,

          i will try it next monday.

          Show
          Thomas Andraschko added a comment - Hi Mark, i will try it next monday.
          Hide
          Mark Struberg added a comment -

          Ok, I think the whole FailOverBag stuff is currently in a bad state. Because it doesn't have a well defined lifecycle and is just too complex for what it does. There are 2 problematic areas

          a.) what you might face: The WebContextsService stores the currently used SessionContext in a ThreadLocal to speed up things. But this ThreadLocal doesn't get set in FailOverBagWrapper#sessionDidActivate as far as I've seen.

          b.) sessionDidActivate() and sessionCreated() are not 100% clearly defined in the Servlet spec. I'll try to ping the EG folks but from just reading the spec I didn't get wise... The actual question is: on a session re-activation we will of course get sessionDidActivate() being invoked. But do we also get sessionCreated()? And in that case do we get it before or after sessionDidActivate()?

          c.) should sessionWillActivate remove the current SessionContext? Remember: we are holding the SessionContext(s) in memory and not in the Session. This is because the Session is x30 times slower than a simple ConcurrentHashMap directly under our control. But if we don't clean up the SessionContext properly, we might end up with wrong contextual instances...

          d.) The current approach is only supporting memcachd backed sessions (no session is held on the server at all). But this is only one of many possible solutions! Another one is full session-replication to all other nodes. In this case the session remains on the server and will in addition gets pushed to all other nodes.

          And there is also the clustering pattern I use in our app (currently with a self-made hack) : We use sticky-session + after the request serialize the Session up to a memcached. And only if the sticky cluster node goes down and the user gets directed to another cluster node, this will detect the missing session and try to restore it from the memcached. This heavily reduces the cost to propagate the sessions after each request (scales linear with each added node and doesn't create square traffic anymore), and also increases the availability: for loosing a session completely you'd need to kill the sticky-node plus the memcached (which can be clustered as well) at the same time! And still then all things will work fine, you'd only get a new session and will be redirected to another cluster node...

          Show
          Mark Struberg added a comment - Ok, I think the whole FailOverBag stuff is currently in a bad state. Because it doesn't have a well defined lifecycle and is just too complex for what it does. There are 2 problematic areas a.) what you might face: The WebContextsService stores the currently used SessionContext in a ThreadLocal to speed up things. But this ThreadLocal doesn't get set in FailOverBagWrapper#sessionDidActivate as far as I've seen. b.) sessionDidActivate() and sessionCreated() are not 100% clearly defined in the Servlet spec. I'll try to ping the EG folks but from just reading the spec I didn't get wise... The actual question is: on a session re-activation we will of course get sessionDidActivate() being invoked. But do we also get sessionCreated()? And in that case do we get it before or after sessionDidActivate()? c.) should sessionWillActivate remove the current SessionContext? Remember: we are holding the SessionContext(s) in memory and not in the Session. This is because the Session is x30 times slower than a simple ConcurrentHashMap directly under our control. But if we don't clean up the SessionContext properly, we might end up with wrong contextual instances... d.) The current approach is only supporting memcachd backed sessions (no session is held on the server at all). But this is only one of many possible solutions! Another one is full session-replication to all other nodes. In this case the session remains on the server and will in addition gets pushed to all other nodes. And there is also the clustering pattern I use in our app (currently with a self-made hack) : We use sticky-session + after the request serialize the Session up to a memcached. And only if the sticky cluster node goes down and the user gets directed to another cluster node, this will detect the missing session and try to restore it from the memcached. This heavily reduces the cost to propagate the sessions after each request (scales linear with each added node and doesn't create square traffic anymore), and also increases the availability: for loosing a session completely you'd need to kill the sticky-node plus the memcached (which can be clustered as well) at the same time! And still then all things will work fine, you'd only get a new session and will be redirected to another cluster node...
          Hide
          Martin Grotzke added a comment -

          The approach you're describing (sticky-session + after the request serialize the session to a memcached etc) is exactly what memcached-session-manager is doing in sticky mode

          Show
          Martin Grotzke added a comment - The approach you're describing (sticky-session + after the request serialize the session to a memcached etc) is exactly what memcached-session-manager is doing in sticky mode
          Hide
          Thomas Andraschko added a comment -

          Should i still test with the newest snapshot? I had no time this week.

          Show
          Thomas Andraschko added a comment - Should i still test with the newest snapshot? I had no time this week.
          Hide
          Mark Struberg added a comment -

          Hi Thomas!

          I'm currently working on improvements. If you could join on irc, then this would help for sure.

          Show
          Mark Struberg added a comment - Hi Thomas! I'm currently working on improvements. If you could join on irc, then this would help for sure.
          Hide
          Mark Struberg added a comment -

          this patch is a first try. It uses the ContextsService instead of the ContextManager natively

          Show
          Mark Struberg added a comment - this patch is a first try. It uses the ContextsService instead of the ContextManager natively
          Hide
          Mark Struberg added a comment -

          I now debugged through and it seems to work. At least with tomcat 7.0.22. I had to disable the org.apache.catalina.mbeans.ServerLifecycleListener because this got removed in tomcat-7.0.9 (see http://tomcat.apache.org/tomcat-7.0-doc/changelog.html)

          Show
          Mark Struberg added a comment - I now debugged through and it seems to work. At least with tomcat 7.0.22. I had to disable the org.apache.catalina.mbeans.ServerLifecycleListener because this got removed in tomcat-7.0.9 (see http://tomcat.apache.org/tomcat-7.0-doc/changelog.html )
          Hide
          Mark Struberg added a comment -

          oki, false positive I fear. It seems that all the OWB SessionContext information gets correctly put into the HttpSession, but the serialisation nor Externalisation doesn't get called. I fear there is something different in tomcat7. Restoring the Session seems ok, but saving the Session to the memcached now happens way before OWB gets the requestDestroyed callback. This is different in tomcat6 though...

          Show
          Mark Struberg added a comment - oki, false positive I fear. It seems that all the OWB SessionContext information gets correctly put into the HttpSession, but the serialisation nor Externalisation doesn't get called. I fear there is something different in tomcat7. Restoring the Session seems ok, but saving the Session to the memcached now happens way before OWB gets the requestDestroyed callback. This is different in tomcat6 though...
          Hide
          Mark Struberg added a comment -

          A short note: if the requestDestroyed is too late, then you could also try the way I use: a simple ServletFilter. I personally do not use the FailOverService at all but implemented an easy solution which grabs the SessionContext via a ServletFilter which also synchronizes on the Session (because of JPA).

          Show
          Mark Struberg added a comment - A short note: if the requestDestroyed is too late, then you could also try the way I use: a simple ServletFilter. I personally do not use the FailOverService at all but implemented an easy solution which grabs the SessionContext via a ServletFilter which also synchronizes on the Session (because of JPA).
          Hide
          Mark Struberg added a comment -

          There is one remaining issue with tomcat7 but this is not related to OWB.
          In tomcat6 and up to tomcat 7.0.21 the StandartEngineValve was being called before SessionTrackerValve.
          With tc-7.0.22 they changed that around thus the contextDestroyed is being called AFTER the session tracker! In OWB or msm we could only provide a workaround with either an own Valve or a ServletFilter. The cleanest way would of course to fix this in tomcat.

          Show
          Mark Struberg added a comment - There is one remaining issue with tomcat7 but this is not related to OWB. In tomcat6 and up to tomcat 7.0.21 the StandartEngineValve was being called before SessionTrackerValve. With tc-7.0.22 they changed that around thus the contextDestroyed is being called AFTER the session tracker! In OWB or msm we could only provide a workaround with either an own Valve or a ServletFilter. The cleanest way would of course to fix this in tomcat.
          Hide
          Mark Struberg added a comment -

          released with OWB-1.1.4

          Show
          Mark Struberg added a comment - released with OWB-1 .1.4
          Hide
          Martin Grotzke added a comment -

          Hi Mark, I'm just trying to reproduce what's going on/wrong here. What's the best way for me to see what changed? Which method in OWB should be called by tomcat (before SessionTrackerValve ends)? I just set a breakpoint to WebBeansConfigurationListener.requestDestroyed but that isn't called at all AFAICS.

          Show
          Martin Grotzke added a comment - Hi Mark, I'm just trying to reproduce what's going on/wrong here. What's the best way for me to see what changed? Which method in OWB should be called by tomcat (before SessionTrackerValve ends)? I just set a breakpoint to WebBeansConfigurationListener.requestDestroyed but that isn't called at all AFAICS.
          Hide
          Mark Struberg added a comment -

          Hi Martin!

          I've debugged this with Thomas already. Here is what we found out so far: It seems that the tomcat team changed the order of the Default Valve and the Session Valve. Up to tomcat-7.0.21 we got the ServletRequestListeners requestDestroyed() BEFORE the msm stores the session to the memcached. Since tomcat-7.0.22 this got changed because the order of the Valves are now different. The requestDestroyed now only gets called after the SessionManager evicts the Session to the memcached. Just set a breakpoint to WebBeansConfigurationListener#requestDestroyed and into your own SessionManager and you will see what I mean.

          Show
          Mark Struberg added a comment - Hi Martin! I've debugged this with Thomas already. Here is what we found out so far: It seems that the tomcat team changed the order of the Default Valve and the Session Valve. Up to tomcat-7.0.21 we got the ServletRequestListeners requestDestroyed() BEFORE the msm stores the session to the memcached. Since tomcat-7.0.22 this got changed because the order of the Valves are now different. The requestDestroyed now only gets called after the SessionManager evicts the Session to the memcached. Just set a breakpoint to WebBeansConfigurationListener#requestDestroyed and into your own SessionManager and you will see what I mean.
          Hide
          Martin Grotzke added a comment -

          > Just set a breakpoint to WebBeansConfigurationListener#requestDestroyed and into your own SessionManager and you will see what I mean.
          I had already set a breakpoint to WebBeansConfigurationListener#requestDestroyed but it was never hit. Do I have to register this listener additionally (in web.xml)?

          Show
          Martin Grotzke added a comment - > Just set a breakpoint to WebBeansConfigurationListener#requestDestroyed and into your own SessionManager and you will see what I mean. I had already set a breakpoint to WebBeansConfigurationListener#requestDestroyed but it was never hit. Do I have to register this listener additionally (in web.xml)?
          Hide
          Mark Struberg added a comment -

          That depends. If you use the openwebbeans-tomcat7.jar plugin as well, then not. Otherwise please add it to your web.xml:
          <listener>
          <listener-class>org.apache.webbeans.servlet.WebBeansConfigurationListener</listener-class>
          </listener>

          Show
          Mark Struberg added a comment - That depends. If you use the openwebbeans-tomcat7.jar plugin as well, then not. Otherwise please add it to your web.xml: <listener> <listener-class>org.apache.webbeans.servlet.WebBeansConfigurationListener</listener-class> </listener>
          Hide
          Martin Grotzke added a comment -

          It seems that my openwebbeans code in eclipse was kind of out of sync with the executed code, that's why the breakpoint was not reached.

          Here are two stacktraces that show the valve ordering and the invocation of ServletRequestListeners (running tomcat 7.0.27):

          Daemon Thread [http-bio-8082-exec-7] (Suspended)	
            ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 184	
            StandardWrapperValve.invoke(Request, Response) line: 225	
            StandardContextValve.invoke(Request, Response) line: 169	
            SessionTrackerValveTC7(SessionTrackerValve).invoke(Request, Response) line: 126	
            NonLoginAuthenticator(AuthenticatorBase).invoke(Request, Response) line: 472	
            StandardHostValve.invoke(Request, Response) line: 168	
            ErrorReportValve.invoke(Request, Response) line: 98	
            StandardEngineValve.invoke(Request, Response) line: 118	
            CoyoteAdapter.service(Request, Response) line: 407
          
          Daemon Thread [http-bio-8082-exec-7] (Suspended)	
            WebBeansConfigurationListener.requestDestroyed(ServletRequestEvent) line: 120	
            StandardContext.fireRequestDestroyEvent(ServletRequest) line: 5970	
            StandardHostValve.invoke(Request, Response) line: 196	
            ErrorReportValve.invoke(Request, Response) line: 98	
            StandardEngineValve.invoke(Request, Response) line: 118	
            CoyoteAdapter.service(Request, Response) line: 407	
            Http11Processor(AbstractHttp11Processor<S>).process(SocketWrapper<S>) line: 999	
            Http11Protocol$Http11ConnectionHandler(AbstractProtocol$AbstractConnectionHandler<S,P>).process(SocketWrapper<S>, SocketStatus) line: 565	
            JIoEndpoint$SocketProcessor.run() line: 307	
            ThreadPoolExecutor$Worker.runTask(Runnable) line: 886	
            ThreadPoolExecutor$Worker.run() line: 908	
            TaskThread(Thread).run() line: 662
          

          So the latter one shows the issue, I'll compare it with tomcat < 7.0.22.

          Show
          Martin Grotzke added a comment - It seems that my openwebbeans code in eclipse was kind of out of sync with the executed code, that's why the breakpoint was not reached. Here are two stacktraces that show the valve ordering and the invocation of ServletRequestListeners (running tomcat 7.0.27): Daemon Thread [http-bio-8082-exec-7] (Suspended) ApplicationFilterChain.doFilter(ServletRequest, ServletResponse) line: 184 StandardWrapperValve.invoke(Request, Response) line: 225 StandardContextValve.invoke(Request, Response) line: 169 SessionTrackerValveTC7(SessionTrackerValve).invoke(Request, Response) line: 126 NonLoginAuthenticator(AuthenticatorBase).invoke(Request, Response) line: 472 StandardHostValve.invoke(Request, Response) line: 168 ErrorReportValve.invoke(Request, Response) line: 98 StandardEngineValve.invoke(Request, Response) line: 118 CoyoteAdapter.service(Request, Response) line: 407 Daemon Thread [http-bio-8082-exec-7] (Suspended) WebBeansConfigurationListener.requestDestroyed(ServletRequestEvent) line: 120 StandardContext.fireRequestDestroyEvent(ServletRequest) line: 5970 StandardHostValve.invoke(Request, Response) line: 196 ErrorReportValve.invoke(Request, Response) line: 98 StandardEngineValve.invoke(Request, Response) line: 118 CoyoteAdapter.service(Request, Response) line: 407 Http11Processor(AbstractHttp11Processor<S>).process(SocketWrapper<S>) line: 999 Http11Protocol$Http11ConnectionHandler(AbstractProtocol$AbstractConnectionHandler<S,P>).process(SocketWrapper<S>, SocketStatus) line: 565 JIoEndpoint$SocketProcessor.run() line: 307 ThreadPoolExecutor$Worker.runTask( Runnable ) line: 886 ThreadPoolExecutor$Worker.run() line: 908 TaskThread( Thread ).run() line: 662 So the latter one shows the issue, I'll compare it with tomcat < 7.0.22.
          Hide
          Martin Grotzke added a comment -

          This is the stacktrace from tomcat 7.0.8:

          (Trying again to get simple formatting right )
          {{
          Daemon Thread ["http-bio-8082"-exec-4] (Suspended (entry into method fireRequestDestroyEvent in StandardContext))
          StandardContext.fireRequestDestroyEvent(ServletRequest) line: 5819
          StandardContextValve.invoke(Request, Response) line: 163
          SessionTrackerValveTC7(SessionTrackerValve).invoke(Request, Response) line: 126
          StandardHostValve.invoke(Request, Response) line: 164
          ErrorReportValve.invoke(Request, Response) line: 100
          StandardEngineValve.invoke(Request, Response) line: 118
          CoyoteAdapter.service(Request, Response) line: 383
          Http11Processor.process(SocketWrapper<Socket>) line: 243
          Http11Protocol$Http11ConnectionHandler.process(SocketWrapper<Socket>, SocketStatus) line: 188
          Http11Protocol$Http11ConnectionHandler.process(SocketWrapper<Socket>) line: 166
          JIoEndpoint$SocketProcessor.run() line: 288
          ThreadPoolExecutor$Worker.runTask(Runnable) line: 886
          ThreadPoolExecutor$Worker.run() line: 908
          TaskThread(Thread).run() line: 662
          }}

          So the difference is that in this version the StandardContextValve.invoke invokes StandardContext.fireRequestDestroyEvent and in tomcat >= 7.0.22 StandardHostValve.invoke does this, right?

          Show
          Martin Grotzke added a comment - This is the stacktrace from tomcat 7.0.8: (Trying again to get simple formatting right ) {{ Daemon Thread ["http-bio-8082"-exec-4] (Suspended (entry into method fireRequestDestroyEvent in StandardContext)) StandardContext.fireRequestDestroyEvent(ServletRequest) line: 5819 StandardContextValve.invoke(Request, Response) line: 163 SessionTrackerValveTC7(SessionTrackerValve).invoke(Request, Response) line: 126 StandardHostValve.invoke(Request, Response) line: 164 ErrorReportValve.invoke(Request, Response) line: 100 StandardEngineValve.invoke(Request, Response) line: 118 CoyoteAdapter.service(Request, Response) line: 383 Http11Processor.process(SocketWrapper<Socket>) line: 243 Http11Protocol$Http11ConnectionHandler.process(SocketWrapper<Socket>, SocketStatus) line: 188 Http11Protocol$Http11ConnectionHandler.process(SocketWrapper<Socket>) line: 166 JIoEndpoint$SocketProcessor.run() line: 288 ThreadPoolExecutor$Worker.runTask(Runnable) line: 886 ThreadPoolExecutor$Worker.run() line: 908 TaskThread(Thread).run() line: 662 }} So the difference is that in this version the StandardContextValve.invoke invokes StandardContext.fireRequestDestroyEvent and in tomcat >= 7.0.22 StandardHostValve.invoke does this, right?
          Hide
          Mark Struberg added a comment -

          Hi!

          Please just set a breakpoint in

          1.) WebBeansConfigurationListener#requestDestroyed()
          This (amongst others) writes the SessionContext (held in RAM due to performance reasons) into the real servlet Session.

          2.) BackupSessionService.backupSession()

          and you will see exactly that they get called in the wrong order since tomcat-7.0.22. I assume it's a bug or misbehaviour in tomcat7 since then.

          What happens since 7.0.22 is that msm first backs up the session, and only later in the request we write the SessionContext into the Session...

          The two important Valves are: StandardEngineValve and SessionTrackerValve which are in different order in tomcat < 7.0.22 and >= 7.0.22.

          Show
          Mark Struberg added a comment - Hi! Please just set a breakpoint in 1.) WebBeansConfigurationListener#requestDestroyed() This (amongst others) writes the SessionContext (held in RAM due to performance reasons) into the real servlet Session. 2.) BackupSessionService.backupSession() and you will see exactly that they get called in the wrong order since tomcat-7.0.22. I assume it's a bug or misbehaviour in tomcat7 since then. What happens since 7.0.22 is that msm first backs up the session, and only later in the request we write the SessionContext into the Session... The two important Valves are: StandardEngineValve and SessionTrackerValve which are in different order in tomcat < 7.0.22 and >= 7.0.22.
          Hide
          Martin Grotzke added a comment -

          Please reread my previous comments including the stacktraces.

          Show
          Martin Grotzke added a comment - Please reread my previous comments including the stacktraces.
          Hide
          Mark Struberg added a comment -

          > So the difference is that in this version the StandardContextValve.invoke invokes
          > StandardContext.fireRequestDestroyEvent and in tomcat >= 7.0.22
          > StandardHostValve.invoke does this, right?

          yes, this observation seems correct.

          Show
          Mark Struberg added a comment - > So the difference is that in this version the StandardContextValve.invoke invokes > StandardContext.fireRequestDestroyEvent and in tomcat >= 7.0.22 > StandardHostValve.invoke does this, right? yes, this observation seems correct.
          Show
          Martin Grotzke added a comment - I just asked on the tomcat dev list: http://tomcat.10.n6.nabble.com/fireRequestDestroyEvent-now-fired-by-StandardHostValve-instead-of-StandardContextValve-td4894573.html

            People

            • Assignee:
              Mark Struberg
              Reporter:
              Thomas Andraschko
            • Votes:
              1 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development