| Summary: | Response mixed between users | ||
|---|---|---|---|
| Product: | Tomcat Connectors | Reporter: | sameem.ahmed |
| Component: | mod_jk | Assignee: | Tomcat Developers Mailing List <dev> |
| Status: | RESOLVED FIXED | ||
| Severity: | critical | CC: | ben, discoversudhir, kjw, p.mouawad, ronanker, sameem.ahmed |
| Priority: | P1 | ||
| Version: | 1.2.28 | ||
| Target Milestone: | --- | ||
| Hardware: | Other | ||
| OS: | Linux | ||
| Bug Depends on: | 57485 | ||
| Bug Blocks: | |||
|
Description
sameem.ahmed
2009-08-20 09:04:36 UTC
AS 4.2.x = jbossweb-2.0.x jbossweb-2.0.x ~ tc-6.0.x the bug could be in mod_jk as well as in the AJP connector, try mod_proxy_ajp if it works then it is most probably a mod_jk bugs otherwise it is in the AJP connector and you could use the jboss forums for help. Has this bug where "responses are being mixed between users" been confirmed in mod_jk? If so, would it apply to all the previous versions of mod_JK? 1.2.21 can exhibit request/response mixing - could be the same issue? One way this can occur is some event that triggers a bug where the http response from tomcat ends up buffered. The next request to that apache worker process will then pick up the previous response from the persistent tcp connection. For this particular type of bug, evidence of an ongoing issue can be seen in a few ways: 1) buffered tcp data between tomcat and apache that stays at the same level - easiest to observe when load is very light - run netstat -a a few times and look for established tcp connections with non 0 data in queues. 2) the responses will be way too fast because they are instant due to previous response already waiting 3) if your using prefork and logging worker process pids, you can look for the pattern by comparing things like response size in logs for URLs served by each pid 4) quiesce the server and then fire a series of requests for a known servlet and verify responses match expected result A workaround or preventative measure is DisableReuse (In reply to comment #0) > Hi, > > We are migrating our application to JBoss container version 4.2.x with Apache > web server in the front. We are using the mod_jk connector version 1.2.28. Our > Apache version is 2.2.6. AJP13 worker is used for communication between web > server and Tomcat. > > We have picked up an issue during load testing where the response is being > mixed between users - one user is able to see reponse of another. Might sound like an dumb question, but what are you load testing against? Have you eliminated the possibility that the application has an error which might cause this? I too happen to notice the same issue on our production boxes with mod_jk-1.2.28, DisableReuse fixed this issue. We are using apache <---> mod_jk-1.2.28 <---AJP13---> tomcat-5.5.23 tomcat logs shows HTTP 302 being returned and apache logs shows HTTP 200 I am trying to reproduce the same on our non-production boxes. The only possible scenario I can think of is the following: * mod_jk: Request 1 sent to tomcat got timeout after 2 mins while waiting for a response * mod_jk : Request 2 reuses the socket of the Request 1 and reads the response of the Request 1 from tomcat. Please let me know if you think the above scenario is not possible. We do have timeouts configured for 2 mins and retries not set(defaults to 2). We did see lots of HTTP 400 error messages(Bad Requests) also caused due to mod_jk and got fixed when we used DisableReuse (In reply to comment #5) Which platform? Are you using tcnative aka the APR connector? What happens if you disable it? > Which platform? Linux (RHEL) > Are you using tcnative aka the APR connector? What happens if you disable it? Yes. I am not able to reproduce this in non-production. Can't try this on production boxes :) --------------------- <Server> <!-- This enables use of the apache portable runtime if apr package is installed --> <Listener className="org.apache.catalina.core.AprLifecycleListener" SSLEngine="on"/> .... </SERVER> --------------------- BTW is there a chance that this can cause server side connection aborts too ? We did see couple of them, not sure if its related to mod_jk. Hi I think we are facing an issue similiar or same as the described and we have also mod_jk 1.2.28. However we can not reproduce it in the test env or during performance testing. Could you give me some hint on how to reproduce the issue you were facing? Is this the scenario to reproduce it? * mod_jk: Request 1 sent to tomcat got timeout after 2 mins while waiting for a response * mod_jk : Request 2 reuses the socket of the Request 1 and reads the response of the Request 1 from tomcat. The thing is that we don't set the socket_timeout (default is infinity) so that does not fit to the described scenario. Is it a true statement that DisableResuse set to false is solving your issue? Cheers Pawel > Is it a true statement that DisableResuse set to false is solving your issue?
Yes, I have not seen another instance after setting this flag on our production boxes.
To follow up Pawels comments in this chain, we went live with DiableReuse recently and it did not resolve the issue for us. We may be looking at a similar issue, but perhaps not this bug. We seem to be experiencing resubmition of POST data, rather than response mixing. Data in a previous unrelated POST request is somehow merged with a subsequent request from a different user. Currently we still do not know the root cause so can not confirm this is a mod_jk issue. It is still critical for us. > We seem to be experiencing resubmition of POST data, rather than response mixing. Just in case you are not aware .... mod_jk retries http post requests in case of timeout in getting response from tomcat with default configurations(no retries or recovery_options specified). See : https://issues.apache.org/bugzilla/show_bug.cgi?id=48400 As part of the DisableReuse, i did set recovery_options to not retry post data and have not seen this issue after that. May be setting both will fix your issue ? For some reason, i was not able to reproduce this behaviour(response swaping) in non-production boxes. (In reply to comment #10) > Data in a previous unrelated POST request is somehow merged with > a subsequent request from a different user. A general comment and not one aimed at your application - the majority of incidents of this i have seen are from application bugs rather than infrastructure ones. It's very easy to mis-share data in a single process, multi-threaded environment. These will often only reveal themselves under truly concurrent request load from multiple login accounts. Perhaps it's worth doing something with userids / unique ids / http headers / logging to allow you to match up requests and responses and verify the correct data is being sent back from the application layer. We have now found out about issues that are resolved in jk .30 that look related, to do with reposting buffers. We did not disable retries. "mod_jk retries http post requests in case of timeout in getting response from tomcat with default configurations(no retries or recovery_options specified)." We will do this anyway, since our application does not like "retries", they can (in theory) result in two or more transactions being processed. I think next test will be with .30 , no reuse, no reties and a new tomcat from JBoss. BTW: We are pretty sure our application is not at fault. I will post results of that test when I get them. # # A fatal error has been detected by the Java Runtime Environment: # # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x48b8e1fe, pid=1412, tid=716 # # JRE version: 6.0_16-b01 # Java VM: Java HotSpot(TM) Client VM (14.2-b01 mixed mode windows-x86 ) # Problematic frame: # C [tcnative-1.dll+0xe1fe] # # If you would like to submit a bug report, please visit: # http://java.sun.com/webapps/bugreport/crash.jsp # The crash happened outside the Java Virtual Machine in native code. # See problematic frame for where to report the bug. # --------------- T H R E A D --------------- Current thread (0x48710400): JavaThread "http-7000-1" daemon [_thread_in_native, id=716, stack(0x4a140000,0x4a180000)] siginfo: ExceptionCode=0xc0000005, reading address 0x00000034 Registers: EAX=0x00000001, EBX=0x00000000, ECX=0x00000000, EDX=0x00000001 ESP=0x4a17ebd8, EBP=0x4a17ebe0, ESI=0x00000000, EDI=0x48710400 EIP=0x48b8e1fe, EFLAGS=0x00010246 Top of Stack: (sp=0x4a17ebd8) 0x4a17ebd8: 48710400 4a17ec1c 4a17ebf8 48b8111f 0x4a17ebe8: 4a17ebf4 00000001 00000000 00000000 0x4a17ebf8: 4a17ec28 00934f8f 48710510 4a17ec1c 0x4a17ec08: 00000001 49d3f6b0 00000000 00000001 0x4a17ec18: 00000001 42b3f598 48710400 48710400 0x4a17ec28: 4a17ed28 007d6e48 49d3f6b0 00000000 0x4a17ec38: 4a17ec5c 48710400 4a17ec5c 6d99918b 0x4a17ec48: 48710400 6d9991b2 42de44b0 48710400 Instructions: (pc=0x48b8e1fe) 0x48b8e1ee: 5d c2 0c 00 83 f8 01 0f 85 80 00 00 00 8b 75 10 0x48b8e1fe: 8b 46 34 85 c0 74 3a 8b 56 14 c7 42 14 10 00 00 Stack: [0x4a140000,0x4a180000], sp=0x4a17ebd8, free space=250k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) C [tcnative-1.dll+0xe1fe] C [tcnative-1.dll+0x111f] J org.apache.tomcat.jni.Address.get(IJ)J J org.apache.coyote.http11.Http11AprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V j org.apache.coyote.Request.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+56 J org.apache.catalina.connector.Request.getRemoteAddr()Ljava/lang/String; J org.apache.catalina.connector.RequestFacade.getRemoteAddr()Ljava/lang/String; J com.ecolog.generic.Session.getSession(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;)Lcom/ecolog/generic/Session; j org.apache.jsp.warehouse_v3.cartitems_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+71 j org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3 J javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V j org.apache.jasper.servlet.JspServletWrapper.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V+395 j org.apache.jasper.servlet.JspServlet.serviceJspFile(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;Ljava/lang/Throwable;Z)V+134 j org.apache.jasper.servlet.JspServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+436 J javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V J org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V J org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V J org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V J org.apache.coyote.http11.Http11AprProcessor.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState; J org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState; j org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+147 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub V [jvm.dll+0xecf9c] V [jvm.dll+0x1741e1] V [jvm.dll+0xed167] V [jvm.dll+0xed1dd] V [jvm.dll+0x116290] V [jvm.dll+0x1d0424] V [jvm.dll+0x173e5c] C [MSVCR71.dll+0xb381] C [KERNEL32.dll+0xb3bc] Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) J org.apache.tomcat.jni.Address.get(IJ)J J org.apache.coyote.http11.Http11AprProcessor.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V j org.apache.coyote.Request.action(Lorg/apache/coyote/ActionCode;Ljava/lang/Object;)V+56 J org.apache.catalina.connector.Request.getRemoteAddr()Ljava/lang/String; J org.apache.catalina.connector.RequestFacade.getRemoteAddr()Ljava/lang/String; J com.ecolog.generic.Session.getSession(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;)Lcom/ecolog/generic/Session; j org.apache.jsp.warehouse_v3.cartitems_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+71 j org.apache.jasper.runtime.HttpJspBase.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+3 J javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V j org.apache.jasper.servlet.JspServletWrapper.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Z)V+395 j org.apache.jasper.servlet.JspServlet.serviceJspFile(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;Ljava/lang/String;Ljava/lang/Throwable;Z)V+134 j org.apache.jasper.servlet.JspServlet.service(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V+436 J javax.servlet.http.HttpServlet.service(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V J org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V J org.apache.catalina.core.ApplicationFilterChain.doFilter(Ljavax/servlet/ServletRequest;Ljavax/servlet/ServletResponse;)V J org.apache.catalina.core.StandardWrapperValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardContextValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardHostValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.valves.ErrorReportValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.valves.AccessLogValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.core.StandardEngineValve.invoke(Lorg/apache/catalina/connector/Request;Lorg/apache/catalina/connector/Response;)V J org.apache.catalina.connector.CoyoteAdapter.service(Lorg/apache/coyote/Request;Lorg/apache/coyote/Response;)V J org.apache.coyote.http11.Http11AprProcessor.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState; J org.apache.coyote.http11.Http11AprProtocol$Http11ConnectionHandler.process(J)Lorg/apache/tomcat/util/net/AprEndpoint$Handler$SocketState; j org.apache.tomcat.util.net.AprEndpoint$Worker.run()V+147 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub --------------- P R O C E S S --------------- Java Threads: ( => current thread ) 0x4906ec00 JavaThread "ajp-8009-1" daemon [_thread_blocked, id=204, stack(0x4c240000,0x4c280000)] 0x4919a800 JavaThread "http-7000-13" daemon [_thread_blocked, id=5484, stack(0x4c1c0000,0x4c200000)] 0x4919a400 JavaThread "http-7000-12" daemon [_thread_blocked, id=5480, stack(0x4c180000,0x4c1c0000)] 0x490ad400 JavaThread "http-7000-11" daemon [_thread_blocked, id=3920, stack(0x4c140000,0x4c180000)] 0x48fc8400 JavaThread "http-7000-10" daemon [_thread_blocked, id=3876, stack(0x4c100000,0x4c140000)] 0x48ebd400 JavaThread "http-7000-9" daemon [_thread_blocked, id=3832, stack(0x4c0c0000,0x4c100000)] 0x48ebcc00 JavaThread "http-7000-8" daemon [_thread_blocked, id=3828, stack(0x4c080000,0x4c0c0000)] 0x49022800 JavaThread "http-7000-7" daemon [_thread_blocked, id=1056, stack(0x4c040000,0x4c080000)] 0x48f24400 JavaThread "http-7000-6" daemon [_thread_blocked, id=584, stack(0x4a800000,0x4a840000)] 0x48b25400 JavaThread "AWT-Windows" daemon [_thread_in_native, id=3640, stack(0x4a7c0000,0x4a800000)] 0x48e7ac00 JavaThread "Java2D Disposer" daemon [_thread_blocked, id=2052, stack(0x4a740000,0x4a780000)] 0x48b39400 JavaThread "http-7000-5" daemon [_thread_blocked, id=976, stack(0x4a700000,0x4a740000)] 0x48aa6400 JavaThread "http-7000-4" daemon [_thread_blocked, id=1696, stack(0x4a6c0000,0x4a700000)] 0x48f26400 JavaThread "http-7000-3" daemon [_thread_blocked, id=1652, stack(0x4a680000,0x4a6c0000)] 0x48afe800 JavaThread "http-7000-2" daemon [_thread_blocked, id=1064, stack(0x4a640000,0x4a680000)] =>0x48710400 JavaThread "http-7000-1" daemon [_thread_in_native, id=716, stack(0x4a140000,0x4a180000)] 0x48e68400 JavaThread "ajp-8009-CometPoller-7" daemon [_thread_blocked, id=3592, stack(0x4a100000,0x4a140000)] 0x48e67000 JavaThread "ajp-8009-CometPoller-6" daemon [_thread_blocked, id=3588, stack(0x4a0c0000,0x4a100000)] 0x48e62400 JavaThread "ajp-8009-CometPoller-5" daemon [_thread_blocked, id=3584, stack(0x4a080000,0x4a0c0000)] 0x48e60c00 JavaThread "ajp-8009-CometPoller-4" daemon [_thread_blocked, id=3304, stack(0x4a040000,0x4a080000)] 0x48e5f800 JavaThread "ajp-8009-CometPoller-3" daemon [_thread_blocked, id=3316, stack(0x4a000000,0x4a040000)] 0x48e5e000 JavaThread "ajp-8009-CometPoller-2" daemon [_thread_blocked, id=640, stack(0x49fc0000,0x4a000000)] 0x48e59400 JavaThread "ajp-8009-CometPoller-1" daemon [_thread_blocked, id=1608, stack(0x49f80000,0x49fc0000)] 0x48e58000 JavaThread "ajp-8009-CometPoller-0" daemon [_thread_blocked, id=1660, stack(0x49f40000,0x49f80000)] 0x48e56800 JavaThread "ajp-8009-Poller-7" daemon [_thread_blocked, id=644, stack(0x49f00000,0x49f40000)] 0x48e55400 JavaThread "ajp-8009-Poller-6" daemon [_thread_blocked, id=632, stack(0x49ec0000,0x49f00000)] 0x48e2c800 JavaThread "ajp-8009-Poller-5" daemon [_thread_blocked, id=1668, stack(0x49e80000,0x49ec0000)] 0x48b53000 JavaThread "ajp-8009-Poller-4" daemon [_thread_blocked, id=636, stack(0x49a40000,0x49a80000)] 0x48b51800 JavaThread "ajp-8009-Poller-3" daemon [_thread_blocked, id=3364, stack(0x49a00000,0x49a40000)] 0x48b50400 JavaThread "ajp-8009-Poller-2" daemon [_thread_blocked, id=3368, stack(0x499c0000,0x49a00000)] 0x48ae2c00 JavaThread "ajp-8009-Poller-1" daemon [_thread_blocked, id=3372, stack(0x49980000,0x499c0000)] 0x48ae1c00 JavaThread "ajp-8009-Poller-0" daemon [_thread_blocked, id=3376, stack(0x49940000,0x49980000)] 0x48b14400 JavaThread "ajp-8009-Acceptor-0" daemon [_thread_blocked, id=3380, stack(0x49900000,0x49940000)] 0x48a66800 JavaThread "http-7000-Sendfile-0" daemon [_thread_blocked, id=3392, stack(0x498c0000,0x49900000)] 0x48a65400 JavaThread "http-7000-CometPoller-7" daemon [_thread_blocked, id=3404, stack(0x49880000,0x498c0000)] 0x489f3c00 JavaThread "http-7000-CometPoller-6" daemon [_thread_blocked, id=3396, stack(0x49840000,0x49880000)] 0x489f2400 JavaThread "http-7000-CometPoller-5" daemon [_thread_blocked, id=3400, stack(0x49800000,0x49840000)] 0x48e50000 JavaThread "http-7000-CometPoller-4" daemon [_thread_blocked, id=3408, stack(0x497c0000,0x49800000)] 0x48e4b800 JavaThread "http-7000-CometPoller-3" daemon [_thread_blocked, id=3412, stack(0x49780000,0x497c0000)] 0x48a62c00 JavaThread "http-7000-CometPoller-2" daemon [_thread_blocked, id=3416, stack(0x49740000,0x49780000)] 0x48a61400 JavaThread "http-7000-CometPoller-1" daemon [_thread_blocked, id=3420, stack(0x49700000,0x49740000)] 0x489e9c00 JavaThread "http-7000-CometPoller-0" daemon [_thread_blocked, id=3428, stack(0x496c0000,0x49700000)] 0x489e8800 JavaThread "http-7000-Poller-7" daemon [_thread_blocked, id=3424, stack(0x49680000,0x496c0000)] 0x48a71400 JavaThread "http-7000-Poller-6" daemon [_thread_blocked, id=908, stack(0x49640000,0x49680000)] 0x48a70400 JavaThread "http-7000-Poller-5" daemon [_thread_blocked, id=3432, stack(0x49400000,0x49440000)] 0x48657800 JavaThread "http-7000-Poller-4" daemon [_thread_blocked, id=3440, stack(0x493c0000,0x49400000)] 0x48656000 JavaThread "http-7000-Poller-3" daemon [_thread_blocked, id=3564, stack(0x49380000,0x493c0000)] 0x4871ec00 JavaThread "http-7000-Poller-2" daemon [_thread_blocked, id=3568, stack(0x49340000,0x49380000)] 0x4871d400 JavaThread "http-7000-Poller-1" daemon [_thread_blocked, id=3560, stack(0x49300000,0x49340000)] 0x48a35c00 JavaThread "http-7000-Poller-0" daemon [_thread_blocked, id=3572, stack(0x492c0000,0x49300000)] 0x489f0400 JavaThread "http-7000-Acceptor-0" daemon [_thread_blocked, id=3576, stack(0x49280000,0x492c0000)] 0x48706800 JavaThread "Thread-1" [_thread_blocked, id=2152, stack(0x48940000,0x48980000)] 0x486cfc00 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=2084, stack(0x488c0000,0x48900000)] 0x486ca400 JavaThread "CompilerThread0" daemon [_thread_blocked, id=1596, stack(0x48880000,0x488c0000)] 0x486c8c00 JavaThread "Attach Listener" daemon [_thread_blocked, id=1928, stack(0x48840000,0x48880000)] 0x486c7800 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=1920, stack(0x48800000,0x48840000)] 0x486b4400 JavaThread "Finalizer" daemon [_thread_blocked, id=1768, stack(0x487c0000,0x48800000)] 0x486b2c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=1764, stack(0x48780000,0x487c0000)] 0x00546400 JavaThread "main" [_thread_in_native, id=1416, stack(0x00440000,0x00480000)] Other Threads: 0x486afc00 VMThread [stack: 0x48740000,0x48780000] [id=1760] 0x486d2800 WatcherThread [stack: 0x48900000,0x48940000] [id=2088] VM state:not at safepoint (normal execution) VM Mutex/Monitor currently owned by a thread: None Heap def new generation total 36352K, used 5720K [0x02600000, 0x04d70000, 0x074c0000) eden space 32320K, 12% used [0x02600000, 0x029c9d60, 0x04590000) from space 4032K, 45% used [0x04980000, 0x04b4c600, 0x04d70000) to space 4032K, 0% used [0x04590000, 0x04590000, 0x04980000) tenured generation total 483968K, used 220495K [0x074c0000, 0x24d60000, 0x42600000) the space 483968K, 45% used [0x074c0000, 0x14c13c50, 0x14c13e00, 0x24d60000) compacting perm gen total 27648K, used 27539K [0x42600000, 0x44100000, 0x46600000) the space 27648K, 99% used [0x42600000, 0x440e4d98, 0x440e4e00, 0x44100000) No shared spaces configured. Dynamic libraries: 0x00400000 - 0x0040f000 C:\Tomcat\bin\tomcat6.exe 0x77f80000 - 0x77ffc000 C:\WINNT\system32\ntdll.dll 0x7c570000 - 0x7c624000 C:\WINNT\system32\KERNEL32.dll 0x77e10000 - 0x77e6f000 C:\WINNT\system32\USER32.dll 0x77f40000 - 0x77f7d000 C:\WINNT\system32\GDI32.dll 0x7c2d0000 - 0x7c335000 C:\WINNT\system32\ADVAPI32.dll 0x77d30000 - 0x77d9f000 C:\WINNT\system32\RPCRT4.dll 0x78000000 - 0x78045000 C:\WINNT\system32\MSVCRT.dll 0x7cf30000 - 0x7d176000 C:\WINNT\system32\SHELL32.dll 0x70a70000 - 0x70ad6000 C:\WINNT\system32\SHLWAPI.dll 0x71710000 - 0x71794000 C:\WINNT\system32\COMCTL32.dll 0x6d800000 - 0x6da8b000 C:\Java\jre6\bin\client\jvm.dll 0x77570000 - 0x775a0000 C:\WINNT\system32\WINMM.dll 0x7c360000 - 0x7c3b6000 C:\WINNT\system32\MSVCR71.dll 0x6d7b0000 - 0x6d7bc000 C:\Java\jre6\bin\verify.dll 0x6d330000 - 0x6d34f000 C:\Java\jre6\bin\java.dll 0x6d290000 - 0x6d298000 C:\Java\jre6\bin\hpi.dll 0x690a0000 - 0x690ab000 C:\WINNT\system32\PSAPI.DLL 0x6d7f0000 - 0x6d7ff000 C:\Java\jre6\bin\zip.dll 0x48b80000 - 0x48c51000 C:\Tomcat\bin\tcnative-1.dll 0x75030000 - 0x75044000 C:\WINNT\system32\WS2_32.dll 0x75020000 - 0x75028000 C:\WINNT\system32\WS2HELP.DLL 0x74ff0000 - 0x75002000 C:\WINNT\system32\MSWSOCK.dll 0x77980000 - 0x779a5000 C:\WINNT\system32\DNSAPI.dll 0x75050000 - 0x75058000 C:\WINNT\system32\WSOCK32.dll 0x7ca00000 - 0x7ca23000 C:\WINNT\system32\rsaenh.dll 0x7ce20000 - 0x7cf0f000 C:\WINNT\system32\ole32.dll 0x7c0f0000 - 0x7c154000 C:\WINNT\system32\USERENV.dll 0x7c740000 - 0x7c7cc000 C:\WINNT\system32\CRYPT32.dll 0x77430000 - 0x77441000 C:\WINNT\system32\MSASN1.dll 0x74fd0000 - 0x74fed000 C:\WINNT\system32\msafd.dll 0x75010000 - 0x75017000 C:\WINNT\System32\wshtcpip.dll 0x6d610000 - 0x6d623000 C:\Java\jre6\bin\net.dll 0x782c0000 - 0x782cc000 C:\WINNT\System32\rnr20.dll 0x77340000 - 0x77353000 C:\WINNT\system32\iphlpapi.dll 0x77520000 - 0x77525000 C:\WINNT\system32\ICMP.dll 0x77320000 - 0x77337000 C:\WINNT\system32\MPRAPI.dll 0x75150000 - 0x75160000 C:\WINNT\system32\SAMLIB.DLL 0x7cdc0000 - 0x7ce10000 C:\WINNT\system32\NETAPI32.DLL 0x7c340000 - 0x7c34e000 C:\WINNT\system32\Secur32.dll 0x77bf0000 - 0x77c01000 C:\WINNT\system32\NTDSAPI.dll 0x77950000 - 0x7797b000 C:\WINNT\system32\WLDAP32.DLL 0x751c0000 - 0x751c6000 C:\WINNT\system32\NETRAP.dll 0x779b0000 - 0x77a4c000 C:\WINNT\system32\OLEAUT32.DLL 0x773b0000 - 0x773df000 C:\WINNT\system32\ACTIVEDS.DLL 0x77380000 - 0x773a3000 C:\WINNT\system32\ADSLDPC.DLL 0x77830000 - 0x7783e000 C:\WINNT\system32\RTUTILS.DLL 0x77880000 - 0x7790e000 C:\WINNT\system32\SETUPAPI.DLL 0x774e0000 - 0x77514000 C:\WINNT\system32\RASAPI32.dll 0x774c0000 - 0x774d1000 C:\WINNT\system32\rasman.dll 0x77530000 - 0x77552000 C:\WINNT\system32\TAPI32.dll 0x77360000 - 0x77379000 C:\WINNT\system32\DHCPCSVC.DLL 0x777e0000 - 0x777e8000 C:\WINNT\System32\winrnr.dll 0x777f0000 - 0x777f5000 C:\WINNT\system32\rasadhlp.dll 0x6d000000 - 0x6d14a000 C:\Java\jre6\bin\awt.dll 0x77800000 - 0x7781e000 C:\WINNT\system32\WINSPOOL.DRV 0x76620000 - 0x76631000 C:\WINNT\system32\MPR.DLL 0x75e60000 - 0x75e7a000 C:\WINNT\system32\IMM32.dll VM Arguments: jvm_args: -Dcatalina.home=C:\Tomcat -Dcatalina.base=C:\Tomcat -Djava.endorsed.dirs=C:\Tomcat\common\endorsed -Djava.io.tmpdir=C:\Tomcat\temp -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager -Djava.util.logging.config.file=C:\Tomcat\conf\logging.properties vfprintf -Xms512m -Xmx1024m java_command: <unknown> Launcher Type: generic Environment Variables: PATH=d:\d2k\bin;C:\WINNT\system32;C:\WINNT;C:\WINNT\System32\Wbem;C:\Program Files\Support Tools\;d:\d2k\jdk\bin OS=Windows_NT PROCESSOR_IDENTIFIER=x86 Family 6 Model 7 Stepping 10, GenuineIntel --------------- S Y S T E M --------------- OS: Windows 2000 Build 2195 Service Pack 4 CPU:total 8 (4 cores per cpu, 1 threads per core) family 6 model 23 stepping 10, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1 Memory: 4k page, physical 2097151k(2097151k free), swap 4194303k(4194303k free) vm_info: Java HotSpot(TM) Client VM (14.2-b01) for windows-x86 JRE (1.6.0_16-b01), built on Jul 31 2009 11:26:58 by "java_re" with MS VC++ 7.1 time: Wed Mar 31 09:08:33 2010 elapsed time: 21683 seconds (In reply to comment #14) > # > # A fatal error has been detected by the Java Runtime Environment: > # I see that the above is also posted separately, as bug 49038. There is no explanation, how that is related to this issue, so I assume this is submitter's fault to post it here. He also changed Product/Component/Version/Platform for this issue. Please ignore comment #14, see bug 49038 if you want to follow it. I am resetting Product etc. to their previous values. FYI We are now running succesfully with modjk .30 , no reuse, no reties, and a new tomcat 4.3.0_CP07, provided by JBoss. This combination seems to be OK, and we have not seen further issues in production systems. I can not confirm that the jk upgrade is what made the difference since we upgraded both modjk and TC at the same time. We were unable to duplicate the issues in test environments. (In reply to Paul Hinds from comment #16) Thank you for follow-up. No reports of such issues for the current version of mod_jk. Closing as WORKSFORME. We are seeing this issue with one of our new products. This product produces many more requests than previous ones. We had two instances this week where a users session was misplaced, or there data files showed up in a list within another users browser. I'm trying to reproduce using a command line tool. I haven't used JMeter. My general conclusion in this area is anything that's effectively doing rpc should be using some sort of id to match requests and responses and allow the client to discard/warn/retry on mismatches. This feature can be found in sun rpc from 1988 and probably earlier implementations, it's not a new concept. This type of bug where the the wrong response ends up with the client and in this case often the web browser can be extremely hazardous. It's a worryingly common bug on connection based transports, pools of connections and multi-threaded clients. (In reply to Kevin J Walters from comment #19) > My general conclusion in this area is anything that's effectively doing rpc > should be using some sort of id to match requests and responses and allow > the client to discard/warn/retry on mismatches. This feature can be found in > sun rpc from 1988 and probably earlier implementations, it's not a new > concept. > > This type of bug where the the wrong response ends up with the client and in > this case often the web browser can be extremely hazardous. It's a > worryingly common bug on connection based transports, pools of connections > and multi-threaded clients. Kevin, Thank you for your advice. Do your represent Apache? I.E. Is this Apache.org's official advice to users? Victor (In reply to Victor from comment #20) > Kevin, > > Thank you for your advice. Do your represent Apache? I.E. Is this > Apache.org's official advice to users? There is no "Apache.org official advice to users." If you wish to discuss this more, please take the discussion to the users' list. It seems the original bug report here has been satisfied. Victor, if you have new evidence that Tomcat and/or mod_jk has a bug, please post them on the users' list and we'll let you know if you should file another bug report. Usually this kind of thing turns out to be a bug in the web application, but it does not rule-out a but in Tomcat/mod_jk. Resetting resolution to FIXED WORKSFORME. This needs to be reopened it happens daily on our production system we are using the latest mod_jk 1.2.40 I also turned off all retrying and connection reuse. switching to mod_proxy resolves the issue. I was unable to reproduce, but describing the scenario seen just in case it helps in reproducing for others ... I had seen multiple instances of this happening(Response of one of the previous request sent to another request) on our production servers. We were sending wrong Content-Length from Tomcat application(More than the actual length) and our tomcat logs showed the correct response size being written to output channel and Apache logs showed less size. Upon looking at Apache logs, saw that this response length and the served content matched with an earlier request. Fixed it by sending correct Content-Length and disabled connection reuse in mod_jk. I found a post about this and the guy was nice enough to post a JMeter script that can reproduce the issue. I have not tested it yet but looks promising. http://jeecookbook.blogspot.com/2013/07/modjk-session-mixed-between-users.html (In reply to Ben Erridge from comment #22) > This needs to be reopened it happens daily on our production system we are > using the latest mod_jk 1.2.40 > > I also turned off all retrying and connection reuse. > > switching to mod_proxy resolves the issue. What version of Tomcat are you using? We are using Glassfish V3.1.2.2 in production. We have the same application running in other datacenters with other Load Balancers, none have exhibited this behaviour except the one fronted by Mod_JK. I have screen shots clearly showing response mix ups. Also because glassfish sends set cookie each response users often take on the identity of other users. This was happening 14-20 times a day. We have a session origin filter which detects when sessions span request IP's but this is not always effective when many users have the same source ip. Once we switched to mod_proxy this never happens. I much prefer mod_jk to mod_proxy so I am happy to provide any details necessary to getting to the bottom of this. I did setup a test platform and make a simple JMeter test but have been unsuccessful in replicating the issue. Our production environment is much more complex than my test environment was. Also I tried to contact Olivier Schmitt who is the gentlemen who has been successful in replicating this in a test environment; But as yet he has not responded to my inquiry about his test rig. (In reply to Ben Erridge from comment #24) > I found a post about this and the guy was nice enough to post a JMeter > script that can reproduce the issue. I have not tested it yet but looks > promising. > > http://jeecookbook.blogspot.com/2013/07/modjk-session-mixed-between-users. > html Hi, the jmeter script (session.bsh) is in fact not correct to detect the session mix as it just compared Request jsessionid cookie et response jsession cookie header setting. But this can be a regular case when during a load test 1 tomcat becomes unresponsive, mod_jk will direct the request to another Tomcat leading to creation of a new JSESSIONID and you end up having a "False Session Mix". Regards Philippe M. (In reply to Ben Erridge from comment #26) > We are using Glassfish V3.1.2.2 in production. We have the same application > running in other datacenters with other Load Balancers, none have exhibited > this behaviour except the one fronted by Mod_JK. > > I have screen shots clearly showing response mix ups. Also because glassfish > sends set cookie each response users often take on the identity of other > users. This was happening 14-20 times a day. We have a session origin filter > which detects when sessions span request IP's but this is not always > effective when many users have the same source ip. > > Once we switched to mod_proxy this never happens. > > I much prefer mod_jk to mod_proxy so I am happy to provide any details > necessary to getting to the bottom of this. > > I did setup a test platform and make a simple JMeter test but have been > unsuccessful in replicating the issue. Our production environment is much > more complex than my test environment was. > > Also I tried to contact Olivier Schmitt who is the gentlemen who has been > successful in replicating this in a test environment; But as yet he has not > responded to my inquiry about his test rig. In fact the blog : http://jeecookbook.blogspot.fr/2013/07/modjk-session-mixed-between-users.html does not really reproduce the case as the way of testing the Mix can trigger false positive as per my previous comment at 2015-03-14 14:04:54 UTC. If you are really able to reproduce it here, it would be nice to provide a full test case with if possible a simple usable application. Thanks Philippe M. Having reviewed this ticket, my working theory is that this was caused by bug 57485. It would explain why it was observed with mod_jk but no mod_proxy. Given that we do not have reproduction steps for this issue I am going to resolve it as fixed. If you still see thins issue - or something like in - with mod_jk 1.2.42 onwards please open a new bug and provide as much of the following as possible: - steps to reproduce from a clean install - network traces (tmpdump, wireshark etc) for the client <-> httpd link and the httpd <-> Tomcat link - mod_jk debug log - Tomcat and httpd access logs - Anything else you think might help to track down the root cause. Hello Mark, What's the expected release date for mod_jk 1.2.42 ? Thanks Possibly as early as next week. Very likely within the next month. Exact timing will depend on other demands on my time. Watch the dev list for more info. |