Bug 54178 - [CVE-2013-2071] runtime exception in onComplete of AsyncListener, will make org.apache.catalina.connector.Request not recycled (orginally reported MESSAGE POST to tomcat, but it called doGet)
Summary: [CVE-2013-2071] runtime exception in onComplete of AsyncListener, will make o...
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 7
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 7.0.23
Hardware: PC All
: P2 normal (vote)
Target Milestone: ---
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-11-21 08:22 UTC by wan_jm
Modified: 2014-02-17 13:39 UTC (History)
0 users



Attachments
picture of the wrong stack catched by wireshark. (187.11 KB, image/jpeg)
2012-11-21 08:23 UTC, wan_jm
Details
tomcat give 500 error when access a static file. (46.43 KB, image/jpeg)
2013-03-18 07:28 UTC, wan_jm
Details
Test case. (2.28 KB, application/x-bzip2)
2013-04-23 08:30 UTC, Leigh Anderson
Details

Note You need to log in before you can comment on or make changes to this bug.
Description wan_jm 2012-11-21 08:22:00 UTC
one HTTP request use POST method, but in tomcat it calls doGet() what is really strage.

as I only tested in 7.0.23, and 7.0.32. in 7.0.32, 7.0.32 is more reproduceable than 7.0.23.

I know it must be a very strange issue, as tomcat has so many release. I can't believe it if I didn't see it with my own eyes.

so I attache the image.
Comment 1 wan_jm 2012-11-21 08:23:46 UTC
Created attachment 29615 [details]
picture of the wrong stack catched by wireshark.
Comment 2 Mark Thomas 2012-11-21 10:46:40 UTC
Filters are perfectly capable of changing the request method. This is an application issue, not a Tomcat issue and the users list is the place to seek help.

Also, if you are going to obscure an IP address, do it properly.
Comment 3 wan_jm 2012-11-22 01:46:32 UTC
thanks for your comments, but most of the time it works properly, just occasionally, it calls the wrong function.

as you said filter may change this. 
I checked the code of the two filter, 

one set the characterEncodeing to UTF-8;

another checcked the cookied whethere there is login information.

no one changed the request method;
Comment 4 Konstantin Kolinko 2012-11-22 09:43:11 UTC
Valves can change a method even easily than filters. E.g. during the FORM authentication (it is not your case, but as example).

1. What is in access log?
2. What is the actual full stacktrace of the error?
3. I think it would be better to discuss on the users@ list, until some way to reproduce the issue is found. Can you reproduce this issue in a simpler configuration?
4. What Connector protocol implementation are you using?
Comment 5 wan_jm 2013-03-18 07:01:54 UTC
I catched the stack from log file, and there is no record in localhost_access*.log; 

following is the stack;
Mar 18, 2013 2:52:48 PM org.apache.catalina.core.StandardWrapperValve invoke
SEVERE: Servlet.service() for servlet [org.apache.vysper.xmpp.extension.xep0124.BoshServlet] in context with path [/LiveVideoServer] threw exception
java.lang.IllegalStateException: Cannot call sendError() after the response has been committed
        at org.apache.catalina.connector.ResponseFacade.sendError(ResponseFacade.java:451)
        at org.apache.vysper.xmpp.extension.xep0124.BoshServlet.doGet(BoshServlet.java:143)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:621)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:722)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:305)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at com.vtc.livemedia.filter.AutoLoginFilter.doFilter(AutoLoginFilter.java:153)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at com.vtc.livemedia.filter.CharactorFilter.doFilter(CharactorFilter.java:48)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:243)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:210)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:224)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:169)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:472)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:168)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:98)
        at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:928)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:118)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:407)
        at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:987)
        at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:539)
        at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:300)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
        at java.lang.Thread.run(Thread.java:722)
Comment 6 wan_jm 2013-03-18 07:22:01 UTC
two things I found strange here!
1. it is the post request, but come to doGet.
2. in the doGet function, we didn't do commit, but the error shows that response has been commited.  in fact there are only two addHeader bofore it.

code is paste here.

  /*line 129*/  @Override
    protected void doGet(HttpServletRequest req, HttpServletResponse resp) throws ServletException, IOException {
        resp.addDateHeader("Date", System.currentTimeMillis());
        resp.addHeader("Server", SERVER_IDENTIFICATION);
        if (FLASH_CROSS_DOMAIN_POLICY_URI.equals(req.getRequestURI())){
            if(accessControlAllowOrigin != null) {
                resp.setContentType(XML_CONTENT_TYPE);
                byte[] flashCrossDomainPolicy = createFlashCrossDomainPolicy();
                resp.setContentLength(flashCrossDomainPolicy.length);
                resp.getOutputStream().write(flashCrossDomainPolicy);
            } else {
                resp.sendError(HttpServletResponse.SC_NOT_FOUND);
            }
        } else {
      /*line 143*/      resp.sendError(HttpServletResponse.SC_METHOD_NOT_ALLOWED, INFO_GET);
        }
        resp.flushBuffer();
    }
Comment 7 wan_jm 2013-03-18 07:28:54 UTC
Created attachment 30063 [details]
tomcat give 500 error when access a static file.

tomcat give 500 internal errors when browser try to download a static file.
but it should give 404 error if the file really not exist;

so I think it has some relationship with another problem I come accross in this thread;

also:
the context is configured in server.xml using follow sentence.

<Context crossContext="true" cachingAllowed="false" docBase="/home/tomcat/userdata/LiveVideoServer/streams" path="/LiveVideoServer/streams" reloadable="true"> </Context>
Comment 8 wan_jm 2013-03-18 07:32:10 UTC
here is the Connector configure.


<Connector executor="tomcatThreadPool" port="8080" protocol="HTTP/1.1"
               connectionTimeout="7200000"
               keepAliveTimeout="7200000"
               maxKeepAliveRequests="-1"
               acceptCount="100"
               enableLookups="false"
               disableUploadTimeout="true"
               maxHttpHeaderSize="8192"
               redirectPort="8443" URIEncoding="UTF-8" />
Comment 9 Mark Thomas 2013-03-19 09:04:59 UTC
Again, this is an application issue not a Tomcat bug. Use the users list.
Comment 10 wan_jm 2013-04-02 09:10:49 UTC
hi Sir, it is reproducible now. after I debug with tomcat source code and my application. first, I will give my analysis; second I will give the opinion that it is a bug, hope I am right; third my question;

first: analysis
  I found that if there is a run-time exception(exclude IOException) in the implication of onComplete, then this exception will be catched in AbstractProtocol$AbstractConnectionHandler.process() in line AbstractProtocol.java:581, and processor is release in line 598 of the same file. 

but the code assume onComplete successfully returned and released org.apache.catalina.connector.Request in function asyncDispatch; So in this scenario Request is not recycled.

when the processor is reused again, the browser request will be the processed by the Request that is not recycled, as there is servlet in it. then the request is handled by the wrong servlet.


second: reason;
  you may tell me that it is the bug of the application; but in fact if this happens in tomcat, request to another application in the same tomcat will got error result it the client request meets the processor which contains the Request that is not recycled;

  I think tomcat should make sure that one application deployed in it should not be interrupt by others? do you think so?

third:
 I want to know why AsyncContextImpl.fireOnComplete catches only IOException instead of Throwable; then maybe the above is fixed;


Sorry for my interrupt.

thanks;
Comment 11 Leigh Anderson 2013-04-23 08:29:14 UTC
I've run into this also. Test case attached -- watch for log messages like

WARNING: Value of test-attribute: test-value

when you browse to /badlyBehaved. I believe that test attribute should never be anything other than null here.
Comment 12 Leigh Anderson 2013-04-23 08:30:57 UTC
Created attachment 30221 [details]
Test case.

Demonstrates that state from the previous request is available in subsequent requests after exception in an AsyncListener.
Comment 13 Mark Thomas 2013-04-24 11:43:35 UTC
(In reply to comment #10)
> hi Sir, it is reproducible now. after I debug with tomcat source code and my
> application. first, I will give my analysis; second I will give the opinion
> that it is a bug, hope I am right; third my question;

Thanks for the additional work to get to the bottom of this.

> first: analysis

The analysis skips over a few stages but is correct. A RuntimeException in a AsyncListener leads to the problems observed because the Request object is not recycled.

> second: reason;

Applications should not be throwing RuntimeExceptions in an AsyncListener but equally it makes sense for Tomcat to protect against that happening in case they do.

> third:
>  I want to know why AsyncContextImpl.fireOnComplete catches only IOException
> instead of Throwable; then maybe the above is fixed;

Only IOException were caught as they were the only ones that should have been thrown. I agree the fix is to catch Throwable (well almost - there are some Throwables that should never be caught but Tomcat has utility code to handle that).

The fix has been applied to trunk and 7.0.x and will be included in 7.0.40 onwards.
Comment 14 Mark Thomas 2013-05-10 08:37:26 UTC
The Tomcat security team has determined that this bug has security implications. It has been assigned CVE-2013-2071. The fix was included in the 7.0.40 release.