Looks like a possible RFC 2616 MUST violation. Apache decodes chunked encoding correctly but forwards wrong Content-Length header which now MUST be honored since Apache removed chunked encoding. Thus, the overall result is that a compliant client would see truncated content (e.g., $100 instead of $1000) because of how Apache changed the message it was forwarding. See attached trace for details and ways to reproduce. Test case IDs in the trace link to human-oriented test case description and RFC quotes, if available.
Created attachment 4352 [details] test case trace
Is the following sentence correct? "A proxy MUST remove a Content-Length header from chunked responses, in case this content length header is wrong".
Not exactly, but very close, IMO. The proxy MUST ignore wrong Content-Length header in chunked responses (which is what Apache is probably doing). It would be OK to forward that header, provided that the proxy does not decode chunks on-the-fly. A compliant recipient would not have a problem handling such a response (chunked encoding with wrong Content-Length header). However, Apache decodes chunks into "normal" (identity) encoding and forwards the response with the original (wrong) Content-Length value. While there is no specific MUST about this case, the only logical interpretation or RFC 2616 suggests that the wrong Content-Length header must be removed. Otherwise, a compliant recipient will mis-interpret the response! If you want a one sentence summary, I would say: "A proxy has to remove the wrong Content-Length header when forwarding a chunked response using identity encoding (i.e., when de-chunking on-the-fly)."
The problem though is that we cannot know the Content-Length is wrong until we have processed the entire request. At this point we would have already sent the Content-Length header onwards, so it's too late to fix it if it is wrong - it would require the proxy to buffer the entire request before sending it on, which is not practical. One possible thing to do though is to remove the Content-Length header should Apache decide to send chunked encoding. This needs to be done inside the filter responsible for chunking, rather than mod_proxy. Thoughts?
The Content-Length header is _always_ wrong when it is used together with chunking because Content-Length header is defined to represent both entity-length and the transfer-length. It is not possible to represent both with a single value when chunking is used. Thus, only two actions would be correct, IMO: - leave wrong Content-Length header and do NOT de-chunk - remove wrong Content-Length header Note that there is no need to remove Content-Length header if Apache decides to send chunked encoding. It is de-chunking that causes the problem.
Is this related to: <http://nagoya.apache.org/bugzilla/show_bug.cgi?id=21348> If so, this bug breaks WindowsUpdate when used behind mod_proxy. Patch is linked in above bug that might be applicable.
The WindowsUpdate patch you mentioned has no visible effect on this bug. Note that the patch seems to be designed to affect HEAD responses while this bug deals with GET responses.
*** Bug 31454 has been marked as a duplicate of this bug. ***
*** Bug 11993 has been marked as a duplicate of this bug. ***
I believe that this problem relates to proxying of _all_ requests with chunked request data, not just those with a bogus Content-Length. mod_proxy is de-chunking and removing Transfer-Encoding: chunked, forwarding a request with neither Content-Length: or Transfer-Encoding: header. This causes the downstream server to reply with error 411 "Length Required". Bug 11618 has a network trace showing this clearly. Since it is not possible to send a correct Content-Length header without buffering all request data, it seems to me that the proper fix is to not de-chunk if the original request is chunked? I looked briefly in the function ap_proxy_http_request in proxy_http.c. How about if the original request uses chunked transfer-encoding, then the proxied request is also send as chunked? I could try to create a patch to add that to the code, it seems not too hard: /* send the request data, if any. */ seen_eos = 0; do { status = ap_get_brigade(r->input_filters, bb, AP_MODE_READBYTES, APR_BLOCK_READ, HUGE_STRING_LEN); if (status != APR_SUCCESS) { return status; } /* If this brigade contain EOS, either stop or remove it. */ if (APR_BUCKET_IS_EOS(APR_BRIGADE_LAST(bb))) { /* As a shortcut, if this brigade is simply an EOS bucket, * don't send anything down the filter chain. */ if (APR_BUCKET_IS_EOS(APR_BRIGADE_FIRST(bb))) { break; } /* We can't pass this EOS to the output_filters. */ e = APR_BRIGADE_LAST(bb); apr_bucket_delete(e); seen_eos = 1; } e = apr_bucket_flush_create(c->bucket_alloc); APR_BRIGADE_INSERT_TAIL(bb, e); status = ap_pass_brigade(origin->output_filters, bb); if (status != APR_SUCCESS) { ap_log_error(APLOG_MARK, APLOG_ERR, status, r->server, "proxy: pass request data failed to %pI (%s)", p_conn->addr, p_conn->name); return status; } apr_brigade_cleanup(bb); } while (!seen_eos);
You do not need to code it as httpd-2.1 has it already fixed. I also backported it in the patch of Bug 31454 which was apparently a DUPE, you can repost the patch here if you wish. I am using my patch on my server for: Fedora Core 3 httpd-2.0.52-3.1
I've been working on some improvements to the current solution in Apache 2.1. A backport of my current 2.0 code is in a patch at http://www.apache.org/~trawick/20proxyreqbody.txt The version of this for Apache 2.1-dev is in the proxy-reqbody branch in subversion. . Any comments or additional testing would be appreciated.
The current version of http://www.apache.org/~trawick/20proxyreqbody.txt successfully tested on SonyEricsson P900 (sorry for the response time as I do not have the device available). <bite>Patch version-name change on its content change would be preferred.</bite>
>Patch version-name change on its content change would be preferred. good idea for the future... and thanks for the test report! I assume that some type of upload from the phone would not work through mod_proxy without this patch????
SonyEricsson P900 uses buzzword WAP-2.0 which no longer transfers WAP/WML over WSP/WTP protocols but it started using HTTP/TCP. P900 will interpret WAP-1.x WAP gateway settings sent over SMS as HTTP proxy for its WAP-2.0 (IMO incorrectly). During MMS send operation P900 will send chunked POST, current mod_proxy will dechunk it but it will not fill in the Content-Length making the final head+body POST request combo invalid. See (I could no longer find the TCP snaps): http://marc.theaimsgroup.com/?l=apache-httpd-dev&m=109186315924289
fix committed to 2.0.next
Created attachment 15295 [details] fresh test trace Sorry, but the current http://www.apache.org/~trawick/20proxyreqbody.txt fix does not solve the problem. Patched Apache httpd-2.0.54 is still sending Content-Length: 15 when the [dechunked] content is actually 20 bytes. Please see the attached fresh trace. Are we testing with the wrong fix?
I am taking the liberty of reopening this bug because the test case still fails (see above).
btw, the proxy-reqbody patch did not affect the response processing... any idea what is required to have the origin server send both Content-Length *and* Transfer-Encoding?
Sending both would violate 2616, if the proxy receives a message with both C-L and T-E it MUST ignore the C-L header per comment 3.
Created attachment 16218 [details] security fix for C-L vs T-E handling (Joe Orton, CVE CAN-2005-2088) Just FYI Joe Orton fixed: Fedora Core Bug: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=162245 , the same as Red Hat Enterprise Bug: https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=162244 but unfortunately these patches will disable the funtionality of: http://www.apache.org/~trawick/20proxyreqbody.txt as presented here in Comment #17 . Reverting its first part restores 20proxyreqbody.txt back as WORKSFORME, Not sure if its second part resolves the issue of Comment #17 .
Is this still open? It appears to be fixed in trunk at least.
OK, tested in 2.2.5; the proxy removes the Content-Length.