Bug 44782 - (104)Connection reset by peer: ap_content_length_filter: apr_bucket_read() failed
Summary: (104)Connection reset by peer: ap_content_length_filter: apr_bucket_read() fa...
Status: RESOLVED INVALID
Alias: None
Product: Apache httpd-2
Classification: Unclassified
Component: All (show other bugs)
Version: 2.2.8
Hardware: PC Linux
: P2 critical (vote)
Target Milestone: ---
Assignee: Apache HTTPD Bugs Mailing List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2008-04-08 21:46 UTC by Tony
Modified: 2008-05-13 12:00 UTC (History)
0 users



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tony 2008-04-08 21:46:12 UTC
After months of trying to figure out what triggers this error:

[Tue Apr 08 18:48:41 2008] [error] [client 123.456.789.012] (104)Connection reset by peer: ap_content_length_filter: apr_bucket_read() failed, referer: http://www.anydomain.com/someform.html

I finally found how to duplicate this on any Apache server. I've found this to be true with all versions from 2.2.4 (earliest 2.x version I have) to 2.2.8 (latest).

Linux CentOS (2.6.18-53.1.14.el5) completely up-to-date. Apache compiled from source. Using mod_perl compiled like this:

shell> perl Makefile.PL MP_USE_STATIC=1 \
MP_AP_PREFIX=/usr/local/src/httpd-2.2.8 \
MP_AP_CONFIGURE="--with-mpm=worker \
--enable-nonportable-atomics=yes \
--prefix=/usr/local/etc/httpd \
--enable-so \
--enable-expires \
--enable-headers \
--enable-rewrite \
--with-ssl=/usr/local/src/openssl-0.9.8g \
--enable-ssl \
--disable-autoindex \
--disable-userdir"

All it takes to produce the above error with Apache is to send a post to a cgi script that does not handle the post data. Example program (test.cgi - not running in mod_perl):

<--snip-->
#!/usr/bin/perl -w
$| = 1;

print "Content-type: text/html\n\n";
print <<"end_of_html";
<html>
<head>
<title>your title</title>
</head>
<body>
anything
</body>
</html>
end_of_html

exit();
<--snip-->

Now setup a simple HTML form and post to the above program:

<--snip-->
<html>
<head>
<title>your title</title>
</head>
<body>
<form action="/cgi-bin/test.cgi" method="post">
<input type="submit" name="test" value="Test It">
</form>
</body>
</html>
<--snip-->

(notice the "name='test'" as some data is required.) The above will cause the browser to hang after post triggering the browser to disconnect from the server and will always produce the above error guaranteed! Change from POST to GET and no "Connection reset by peer" error. The test.cgi script will exit properly, but Apache keeps the connection alive and then the browser disconnects.

I have tested this with Linux Firefox/1.5.0.12 - 2.0.0.13, Opera 9.26 as well as on Windows XP Professional with Firefox/2.0.0.13 and IE 7.0.5730.11.

Note: If you add this to the top of the test.cgi program:

read(STDIN, my $buffer, $ENV{'CONTENT_LENGTH'});

No disconnect is encountered.

I believe this to be an Apache problem as all earlier versions of Apache did not behave in this manner.
Comment 1 Tony 2008-05-02 22:06:57 UTC
Is there a point in posting? Doesn't seem anything, not even a comment about this. Is it not worthy?
Comment 2 Vinci 2008-05-04 11:42:46 UTC
I encounter same bug with mod_perl 2.0.2 (perl 5.8.8), on apache 2 running at worker mode (apache2-mpm-worker) 2.2.4 on Ubuntu 7.10. Everything come from apt-get.

Fortunately, if you using GET at all the time you won't notice by this bug  

The solution works but this is not graceful if I need to do that on all cgi-script that receive POST request.

Actually this bug will causing unexpected 500 error when running with mod_security. 
Comment 3 Tony 2008-05-08 00:48:48 UTC
Yes this is a big bug and I don't know why no one from Apache has even commented on this post. My temporary solution so far is to add this line to programs before exit:

if ($ENV{'CONTENT_LENGTH'}) {while (<STDIN>){}}

Vinci, you state it is not a problem with GET, but the problem is mainly if you use "GET" in your application and do not write for a POST. Some bots out there may try using your GET method forms/links, but instead may send a POST and that's where the problem is. Plus bots or any malicious user could try sending HUGE amounts of data via a POST bringing your server down as Apache waits for the POST data to be read. My solution to prevent this has been to limit how much of the $ENV{'CONTENT_LENGTH'} I would read in, but with this bug it just hangs your server until the client has finished sending their POST. The only solution I've been able to come up with is the one above, but it is band-aide rather than a fix. I sure wish someone from Apache would at least comment on this.
Comment 4 Ruediger Pluem 2008-05-08 04:19:26 UTC
This works as designed. It is the task of the cgi script to swallow the whole request body prior to sending a response, whether it uses it for some purpose or not.
Comment 5 Tony 2008-05-13 08:45:43 UTC
Of course this "as designed" is not good. You are asking everyone to modify everything they've done in past years to accept and "gobble up" all incoming "request body" (post, get, head and ?) content even though their programs may not be parsing incoming data. 

Some programs are designed to do nothing more but fetch a static document or print an image or ? and be triggered by nothing more than the execution of their script. Many programs may do nothing more than generate a cookie. But you are now asking us to create Microsoft bloatware to handing incoming data by either including our own routines to gobble things up (any kind of request) or include libraries (id, like Perl use CGI) in our code to create bloatware so that Apache terminates the "bad" connection! NOT GOOD!

Personally I do not agree with this as designed. This is obviously a change that either I missed or was not documented in the changes. As far as I know this was never a problem in past versions of Apache. 

With the growing number of worms and viruses, developers would rather NOT "gobble up" incoming data if our program is not going to use it. Nor do we want to include libraries or write code to gobble every possible type of incoming data. Apache should accept our "exit" code from our program and terminate the connection between client and server regardless of the reason for our "exit" code. By keeping connections between client and server open after our exit you are asking for trouble. The more worms hitting the server that you keep active the more vunerable our Apache server is. You are asking us to re-develop everything we've done to handle all or any type of incoming data. For me I feel this is a burden you are asking us to take on.

Just my opinion of this which I'm sure others share.
Comment 6 Ruediger Pluem 2008-05-13 12:00:48 UTC
Sorry but this behaviour is there for ages (at least since 2.0.x) and it does not put a large burden on the cgi script developer. Simply read all the data from standard input and throw it away immediately without any further handling or just close the stdin file handle in your cgi script and you are done. If you close the stdin filehandle in your cgi script httpd will take the task and the burden to swallow all incoming data and discard it.
If you don't like doing this configure you cgi scripts via <limit> to only accept GET requests.