On my server, I recently added the DEFLATE filter for all application/x-javascript files, and noticed that apache has stopped sending 304 NOT MODIFIED on ALL my js files *always*. Observe this header trace: ----------------------------------------- GET /main.js HTTP/1.1 Host: xxxxxxxxxxxxx User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.1.14) Gecko/20080404 Firefox/2.0.0.14 Accept: */* Accept-Language: en-us,en;q=0.5 Accept-Encoding: gzip,deflate Accept-Charset: ISO-8859-1,utf-8;q=0.7,*;q=0.7 Keep-Alive: 300 Connection: keep-alive If-Modified-Since: Mon, 12 May 2008 16:09:00 GMT If-None-Match: "44d0ac3fd1f00"-gzip Cache-Control: max-age=0 HTTP/1.x 200 OK Date: Sat, 17 May 2008 06:09:25 GMT Server: Apache Last-Modified: Mon, 12 May 2008 16:09:00 GMT Etag: "44d0ac3fd1f00"-gzip Accept-Ranges: bytes Cache-Control: max-age=86400, must-revalidate, private Expires: Sun, 18 May 2008 06:09:25 GMT Vary: Accept-Encoding Content-Encoding: gzip Content-Length: 9797 Keep-Alive: timeout=2, max=299 Connection: Keep-Alive Content-Type: application/x-javascript ----------------------------------------- It seems to me, that a 304 NOT MODIFIED would have been the appropriate response here. Before I added DEFLATE to javascript files, the above response was always 304 NOT MODIFIED (except the first one). Now with deflate, it is *always* 200 OK. Is this by design, or is this a bug? One thing to mention is - I have both mod_deflate and mod_expires statically compiled into httpd.
If we remove "-gzip" from Etag for If-None-Match, a server returns 304. mod_deflate adds "-gzip" and sends it to us, but condition check of If-None-Match runs before mod_deflate ...???
This is actually a fix of Bug 39727. Yes, we know the core code testing for conditional responses still needs work.
This bug nearly defeats the purpose of mod_deflate. I turned mod_deflate on with the intent of saving bandwidth when sending JavaScript and CSS files. Instead, I'm using *more* bandwidth for every request but the first. (Previously, subsequent requests would result in small, no-content 304 responses. Now the compressed JS and CSS files are sent every time.) BTW, the change that appends "-gzip" to the etag (revision 607219, I think) is a bit wacky in its own right in that it appends outside the double quotes. IOW, if the original header is: Etag: "5954c6-10f4-449d11713aac0" the modified header ends up as: Etag: "5954c6-10f4-449d11713aac0"-gzip when it probably should be: Etag: "5954c6-10f4-449d11713aac0-gzip" with the "-gzip" inside the quotes. I altered the code to do this, but it had no affect on this 304 bug. I just wanted to note it here so the situation is avoided when this bug is fixed for real.
There is a workaround, If you are serving from a location say /js You can use a configuration like below to switch ETag: $1-gzip to $1 <Location /js> RequestHeader edit "If-None-Match" "^(.*)-gzip$" "$1" Header edit "ETag" "^(.*[^g][^z][^i][^p])$" "$1-gzip" </Location> (Perhaps this should be done from mod_deflate as input filter with the condition that if Client requests with Accept-Encoding: gzip only then modify If*match headers to their un-gzip values.) (Also the current value of ETag (+gzip) is not RFC compliant, since RFC mandates a quoted string as the value of ETag. As the noted by john, -gzip should be inside quotes.)
Created attachment 22453 [details] A POC slightly better than the above conf lines, Adds an input filter ETAG that can be used like below, along with DEFLATE to let apache recognize ($1)-gzip as $1 AddInputFilter ETAG .txt AddOutputFilterByType DEFLATE text/plain
Both the workaround in comment #4 and the patch in comment #5 appear to work. Neither are real "fixes" for the bug, however. (But I'm glad I have something to work with until a real fix arrives, so thanks rahul!)
Created attachment 22468 [details] use ap_hook_post_read_request instead of input filter Incorporate suggested changes by Nick
Hello! I'm also facing the same problem -- I had been scratching my head for sometime before I found this bug report. I have to reduce my page size urgently and not having gzip on is not really an option for me. I'm not really comfortable having a customized compiled version of apache2 on my system and the regular expression "hack" will also slow things down (will it?). Any alternative ideas? When might this bug be fixed? Thanks, Sidharth
I had a look at ap_hook_post_read_request attachment...seems like I can get away with just compiling mod_deflate.c I'm using Apache 2.2.8. Can anyone help me on where I can get the correct source for the corresponding mod_deflate and how I can compile mod_deflate? Thanks for your help! Sidharth
Hi, Sorry for flooding this mailing list. In this thread there is Rahul's mod_deflate patch and there are some patches in this bug thread: https://issues.apache.org/bugzilla/show_bug.cgi?id=39727 (mostly by Nick Kew) Which one is the best solution? Thanks, Sidharth
You will need both. This patch expects the other to have been applied (this is already applied on trunk). Also please use the dev@httpd.apache.org for questions.
Thanks for your response Rahul. Unfortunately I am new to all of this. I applied Nick's patch to my mod_deflate.c but I got patching errors...his patch is assuming a slightly different mod_deflate.c from mine. Would really appreciate if you were able to direct me to the place (or even better just gave me the patched mod_deflate.c version) that I could just compile. [ASIDE: I am using apxs2 -c mod_deflate.c with LDFLAGS="-lz" set in /usr/bin/aprconfig I notice that my original mod_deflate.so depends on libpthread. When I apply your patch to my file and compile, I don't get dependency with libpthread.] Please tell me if I would need to do any additional transformations to the file you would provide me (assuming you can :-) ). Thanks a Ton! Sidharth [Attaching the mod_deflate that I have. This I obtained by issuing sudo apt-get install apache2-src on ubuntu. This is version 2.2.8 Apache]
Created attachment 22605 [details] mod_deflate as found in apache2-src package on ubuntu version 2.2.8-1ubuntu0.3
This bug was added in 2.2.8 in a failed attempt to address Bug 39727. Since a real fix will require extensive changes, I have reverted the change in http://svn.apache.org/viewvc?view=rev&revision=761835 Note: if you are patching a released version of the source code, then extract the patch from the original change http://svn.apache.org/viewvc?view=rev&revision=608849 and use patch -R to apply it in reverse. Meanwhile, the original bug will still be tracked as issue 39727.
I'm still experiencing this with 2.2.11-2ubuntu2 -- but adding these 2 lines are suggested by rahul works: RequestHeader edit "If-None-Match" "^(.*)-gzip$" "$1" Header edit "ETag" "^(.*[^g][^z][^i][^p])$" "$1-gzip"
Was this ever resolved in trunk/2.4? Alias /deflate /home/covener/SRC/httpd-trunk/built/htdocs/index.html <Location /deflate> SetOutputFilter DEFLATE </location> $ wget --header="Accept-Encoding: gzip" -S http://localhost/deflate -O/dev/null 2>&1 |grep ETag ETag: "58c5-4b26c6f28ce80-gzip" $ wget --header='If-None-Match: "58c5-4b26c6f28ce80-gzip"' --header="Accept-Encoding: gzip" -S http://localhost/deflate -O/dev/null 2>&1 --2013-06-23 13:39:47-- http://localhost/deflate Resolving localhost (localhost)... 127.0.0.1 Connecting to localhost (localhost)|127.0.0.1|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Date: Sun, 23 Jun 2013 17:39:47 GMT Server: Apache/2.5.0-dev (Unix) OpenSSL/1.0.1c Last-Modified: Wed, 23 Nov 2011 20:04:58 GMT ETag: "58c5-4b26c6f28ce80-gzip" Accept-Ranges: bytes Vary: Accept-Encoding Content-Encoding: gzip Content-Length: 206 Keep-Alive: timeout=5, max=100 Connection: Keep-Alive Content-Type: text/html covener@cov-w520:~/SRC/httpd-trunk$ wget --header='If-None-Match: "58c5-4b26c6f28ce80"' --header="Accept-Encoding: gzip" -S http://localhost/deflate -O/dev/null 2>&1|grep HTTP/1.1 HTTP/1.1 304 Not Modified
It's a copout, but I've made this configurable in r1586542 with a default for now of maintaining the 2.4 behavior.
*** Bug 56354 has been marked as a duplicate of this bug. ***
Eric, I am graceful for the new directive, but why not implementing a better ETAG verification instead of altering the ETAG generation?. If you get an etag with a "-gzip" suffix, you just verify the etag without that suffix. You could link this to the "accept-encoding" header, maybe.
(In reply to Jesús Cea from comment #19) > Eric, I am graceful for the new directive, but why not implementing a better > ETAG verification instead of altering the ETAG generation?. My immediate concern is the unnecessary difference between 2.2 and 2.4, allowing the 2.2 behavior in 2.4 is a lot simpler to tackle then adding some third behavior (I admitted before it was a copout)
(In reply to Eric Covener from comment #17) > It's a copout, but I've made this configurable in r1586542 with a default > for now of maintaining the 2.4 behavior. Hi Eric, will your workaround of r1586542 make it into some Apache 2.4 release? I saw that it is integrated in 2.5, but it is not mentioned in the 2.4 docs. I tried the DeflateAlterETag directive, but it does not seem to available in Ubuntu 14.04 LTS (Apache 2.4.7). Since you only added a directive, this cannot break any existing server configurations, so it should be available in 2.4, too!
Am I correct to read in http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.26 that the If-None-Match header can contain mulitple ETAG values? If so, maybe an alternative version of rahuls workaround could be: RequestHeader edit "If-None-Match" '^"((.*)-gzip)"$' '"$1", "$2"' With above line, I think there is no need to modify the outgoing Header and the modefied RequestHeader works in with or without deflation
Undo spam change
Undo spam change.
(In reply to Martin Heide from comment #21) > (In reply to Eric Covener from comment #17) > > It's a copout, but I've made this configurable in r1586542 with a default > > for now of maintaining the 2.4 behavior. > > Hi Eric, > will your workaround of r1586542 make it into some Apache 2.4 release? I saw > that it > is integrated in 2.5, but it is not mentioned in the 2.4 docs. > I tried the DeflateAlterETag directive, but it does not seem to available in > Ubuntu 14.04 LTS (Apache 2.4.7). > > Since you only added a directive, this cannot break any existing server > configurations, so it should be available in 2.4, too! *tap*tap*tap*, is this thing on? Why do we have BrotliAlterETag in 2.4, but DeflateAlterETag not and is r1586542 still waiting to be backported?
With a minor modification, the workaround in Comment 22 works for both mod_deflate and mod_brotli : RequestHeader edit "If-None-Match" '^"((.*)-(gzip|br))"$' '"$1", "$2"' (using the default AddSuffix setting for both modules' ETag handling). Seems to me that this should just be incorporated into the "If-None-Match"-handling code in the server, rather than relying on users to monkey with their httpd.config file. But what do I know ?
I think I'm seeing the same issue with * Apache 2.4.39 * HTTP/2 * GZipped transfer When I first retrieve the URL, I get an ETag with '-gzip' appended. $ curl -I 'https://example.net/file.png' --compressed HTTP/2 200 date: Fri, 02 Aug 2019 00:12:51 GMT server: Apache/2.4.39 (Unix) LibreSSL/2.9.1 mod_chroot/0.5 last-modified: Fri, 02 Aug 2019 00:08:49 GMT etag: "11b104-4fa0-58f1729a20024-gzip" accept-ranges: bytes cache-control: max-age=31536000 expires: Sat, 01 Aug 2020 00:12:51 GMT vary: Accept-Encoding content-encoding: gzip strict-transport-security: max-age=15768000; content-length: 18360 content-type: image/png Any subsequent requests for that full ETag result in a 200 with the data being re-transfered. $ curl -I 'https://example.net/file.png' --compressed -H 'If-None-Match: "11b104-4fa0-58f1729a20024-gzip"' HTTP/2 200 date: Fri, 02 Aug 2019 00:12:55 GMT server: Apache/2.4.39 (Unix) LibreSSL/2.9.1 mod_chroot/0.5 last-modified: Fri, 02 Aug 2019 00:08:49 GMT etag: "11b104-4fa0-58f1729a20024-gzip" accept-ranges: bytes cache-control: max-age=31536000 expires: Sat, 01 Aug 2020 00:12:55 GMT vary: Accept-Encoding content-encoding: gzip strict-transport-security: max-age=15768000; content-length: 18360 content-type: image/png If I remove the '-gzip' suffix, I get the expected 304 response. $ curl -I 'https://example.net/file.png' --compressed -H 'If-None-Match: "11b104-4fa0-58f1729a20024"' HTTP/2 304 date: Fri, 02 Aug 2019 00:12:58 GMT server: Apache/2.4.39 (Unix) LibreSSL/2.9.1 mod_chroot/0.5 etag: "11b104-4fa0-58f1729a20024" expires: Sat, 01 Aug 2020 00:12:58 GMT cache-control: max-age=31536000
The updated solution from Comment 26 fixed the issue (didn't try Comment 22).
the issue with Comment 26 seems to me that if you send a request with `if-none-match: "0-5debc62fd30dd-br"` it'll respond with `etag: "0-5debc62fd30dd"` (which is missing the `-br` part) it seems to me that something like the following solves it SetEnvIf If-None-Match '^"((.*)-(gzip))"$' gzip SetEnvIf If-None-Match '^"((.*)-(br))"$' br RequestHeader edit "If-None-Match" '^"((.*)-(gzip|br))"$' '"$1", "$2"' Header edit "ETag" '^"(.*)"$' '"$1-gzip"' env=gzip Header edit "ETag" '^"(.*)"$' '"$1-br"' env=br