Bug 37145

Summary: data loss with httpd-2.0.55 reverse proxy method=post
Product: Apache httpd-2 Reporter: peter.schlaf
Component: mod_proxyAssignee: Apache HTTPD Bugs Mailing List <bugs>
Status: RESOLVED FIXED    
Severity: critical CC: apache, aragon, arjan.schaaf, baptiste.desprez, dbergstr, elwood, erik.wasser, girish.nair1, jgauthier, lswick, marco.casavecchia, ogee, shien.xie, stijn
Priority: P1 Keywords: PatchAvailable
Version: 2.0.55   
Target Milestone: ---   
Hardware: PC   
OS: Linux   
Attachments: Patch against 2.0.55
Fix 3 addtional occurences. Patch against 2.0.55

Description peter.schlaf 2005-10-18 15:37:56 UTC
2 PCs with httpd-2.0.55 running and https enabled (OS SLES9). The first one is
the "real" webserver with all the documents (HTML-pages, etc), the other
webserver is configured as a reverse proxy which gets all the data from the
"real" webserver.

With a browser I can access both webservers and everything seems to be OK.

The problem: 
When I get a form (method="post") through the reverse proxy and send it back,
some of the "post"ed data dont reach the "real" webserver.
It seems to happen always if the total amount of posted data is larger than 8
kbytes.

When I get a form from the "real" webserver directly and send it back there is
no loss of data (at any amount).

The "real" webserver and the reverse proxy are talking via https to each other.
Switching to http does not solve the problem.

When _both_ webservers are switched to http the problem has disappeared, but
this is not really a solution for me.

Downgrading the reverse proxy to httpd-2.0.54 (both webservers with https,
"real" webserver stays with httpd-2.0.55) solved the problem. 

During all tries I left the httpd.conf and ssl.conf unchanged (exception:
ProxyPass and ProxyPassReverse for http/https switching of course)

Reverse proxy configuration:
<IfModule mod_proxy.c>
    SSLProxyEngine       on
    ProxyRequests        off
    ProxyPass            /  https://10.34.56.78/
    ProxyPassReverse     /  https://10.34.56.78/
</IFModule>
Comment 1 Ruediger Pluem 2005-10-18 21:38:18 UTC
Thanks for the report. Some questions:
1. Could you please attach an errorlog for a problematic POST request with
LogLevel set to debug?
2. You say that some of the data posted gets lost when the posted data is larger
than 8k. Can you specify how much data gets lost (e.g. Does everything beyond
the first 8k get lost or are only the last 8k missing or something else)?
Comment 2 Ruediger Pluem 2005-10-18 22:08:56 UTC
Please enable also mod_dumpio on the reverse proxy,
set

DumpIOInput On

in your configuration and attach the output for a faulty POST request (contained
in errorlog file).
Comment 3 Ruediger Pluem 2005-10-18 23:44:51 UTC
After a short discussion on IRC with William A. Rowe, Jr. (a.k.a. wrowe,
OtherBill), he additionally proposed to add the following patch to your reverse
 proxy and check if the problem is gone then:
http://people.apache.org/~wrowe/httpd-2.0-proto-timeout.patch
Comment 4 Ruediger Pluem 2005-10-19 03:00:47 UTC
As of now I guess you can wait with answering my questions / adding OtherBills
patch. I already have an idea what the reason is and need to discuss this with
the other developers on the dev list.
Comment 5 peter.schlaf 2005-10-19 10:15:23 UTC
The patch was applied and didn't solve the problem.

In a form with a lot of <input type="hidden" ..> lines only the _last_ approx. 8
kbytes reach the "real" webserver (visible by dumping the $_POST hash in the
"action" PHP-script). All data posted before are lost.

Using the form content <input type="file" ..> I can only upload files smaller
than 7660 bytes. Otherwise I get an error-message of the "action"-script
complaining about missing parameters.
Comment 6 Ruediger Pluem 2005-10-19 11:07:51 UTC
Created attachment 16743 [details]
Patch against 2.0.55

Could you please give the attached patch a try (on the reverse proxy)? I have
not finished my discussion on possible sideeffects with other developers, so
the patch might be subject to change. But it would be helpful to know if it
fixes your problem.
Comment 7 peter.schlaf 2005-10-19 11:21:26 UTC
Your attached patch solved the problem. :-)

Thank you very much!
Comment 8 Ruediger Pluem 2005-10-19 12:29:01 UTC
Created attachment 16744 [details]
Fix 3 addtional occurences. Patch against 2.0.55

Can you please also test the following extended patch? Meanwhile I identified 3
others positions in the proxy code path which can cause this kind of trouble.
Would be nice if you can give it a try too :-)
Comment 9 peter.schlaf 2005-10-19 13:07:43 UTC
(In reply to comment #8)
> Fix 3 addtional occurences. Patch against 2.0.55

This patch is working fine either. The reverse proxy performs perfectly. No loss
of data anymore.
Comment 10 Ruediger Pluem 2005-10-19 14:22:37 UTC
Thanks for the fast feedback.
Comment 11 Helmut Schneider 2005-10-24 19:31:20 UTC
(In reply to comment #10)
> Thanks for the fast feedback.

JFYI, this patch is also working fine with Outlook Web Access on IIS6 as proxied
machine.
Comment 12 Ruediger Pluem 2005-10-24 20:41:34 UTC
Meanwhile the patch was commited to trunk (r327590):
http://svn.apache.org/viewcvs?rev=327590&view=rev 

To 2.2.x (r327592):
http://svn.apache.org/viewcvs?rev=327592&view=rev

and proposed for backport to 2.0.x:
http://svn.apache.org/viewcvs?rev=327597&view=rev
Comment 13 desprez 2005-10-26 13:15:29 UTC
*** Bug 37251 has been marked as a duplicate of this bug. ***
Comment 14 Ruediger Pluem 2005-11-22 22:06:21 UTC
*** Bug 37421 has been marked as a duplicate of this bug. ***
Comment 15 Marco Casavecchia 2005-12-07 12:21:39 UTC
*** Bug 37739 has been marked as a duplicate of this bug. ***
Comment 16 Ruediger Pluem 2006-01-05 21:10:42 UTC
*** Bug 38141 has been marked as a duplicate of this bug. ***
Comment 17 ApacheWorker 2006-01-19 00:44:23 UTC
Any timeframe on when 2.0.56 will be out with the patch in it?

Lorinda
Comment 18 Ruediger Pluem 2006-02-04 11:20:45 UTC
The patch has been backported to 2.0.x r372046
(http://svn.apache.org/viewcvs?rev=372046&view=rev) and will be part of 2.0.56.
For the status of 2.0.56 please have a look at
http://svn.apache.org/viewcvs.cgi/httpd/httpd/branches/2.0.x/STATUS?view=markup.
Comment 19 Ruediger Pluem 2006-03-03 15:14:41 UTC
*** Bug 38837 has been marked as a duplicate of this bug. ***
Comment 20 Ruediger Pluem 2006-03-15 09:09:49 UTC
*** Bug 38974 has been marked as a duplicate of this bug. ***
Comment 21 Ruediger Pluem 2006-04-11 22:22:37 UTC
*** Bug 39274 has been marked as a duplicate of this bug. ***
Comment 22 Nick Kew 2006-05-08 00:28:22 UTC
*** Bug 38505 has been marked as a duplicate of this bug. ***
Comment 23 Alex Georgopoulos 2006-06-29 00:30:22 UTC
I am still seeing these errors on 2.0.58 & 2.2.2 using mod_proxy for a reverse
proxy for RPC over HTTP and Exchange.  Downgrading to 2.0.54 fixes the problem,
maybe there are still some other occurences besides the ones patched here that
have not been fixed?  I also tried this on 2.2.2 and it didn't work either.

[Tue Jun 27 18:08:08 2006] [error] (70014)End of file found: proxy: prefetch
request body failed to 10.2.181.53 from 15.235.153.107 ()
[Tue Jun 27 19:07:46 2006] [error] (104)Connection reset by peer: proxy:
prefetch request body failed to 10.2.181.53 from 10.2.181.18 ()
Comment 24 Ruediger Pluem 2006-06-29 09:34:57 UTC
Sorry, but currently I do not see a relation to this bug. Your error messages
indicate that the connection to the client broke during reading the POST request
body. This bug is about POST request bodies passed to the backend server getting
corrupted.