PUT to an unmapped URL (DELETEd before) with If-None-Match: * currently fails with 412, but should succeed (see <http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.26.p.8>).
It seems that If-Match and If-None-Match in general aren't evaluated properly when the target does not exist. For instance: LOCK /unmapped If-Match: "*" should fail with 412, but creates the lock. Or LOCK /unmapped If-None-Match: "*" should succeed, but fails with 412.
Created attachment 19658 [details] test cases for LOCK request with if-* headers
Isn't there any programmer somewhere to help fix this bug. It is a *severe* bug in mod_dav and it is open for at least four years, while IIS does it right.
Created attachment 20552 [details] Fixes If-Match: * and If-None-Match: * bug for mod_dav The bug is in function ap_meets_conditions() in modules/http/http_protocol.c: it always evaluates "If-Match: *" to TRUE (is FALSE, if resource does not exist) and "If-None-Match: *" to FALSE (is TRUE, if the resource does not exist). This function is called by mod_dav, function dav_validate_request(). In this case, ap_meets_conditions() seems not able to get etag reliably (probably a bug in mod_dav). Fix: A new function dav_meets_conditions() is created in modules/dav/main/util.c. It is mostly a copy of ap_meets_conditions(), but fixes the mentioned errors. dav_validate_request() calls dav_meets_conditions() instead of ap_meets_conditions(). ToDo: it would be better to fix this in ap_meets_conditions(). But to do this, this functions must know, whether the resource exists, and it must be able to reliably get the etag of the resource. But as I am not familiar with Apache-programming, I can't do this. I even doubt that it is possible without changes in other Apache modules besides mod_dav.
Created attachment 20553 [details] Test results for conditional LOCK-requests HTTP-body and irrelevant headers are removed. Clients should remove "W/" from weak Etags. Apache always creates weak Etags, when a request is sent within less than 1 second after the last modification. So clients that use HEAD to get the Etag immediately after PUT will be fooled when they use this Etag some seconds later.
Why would a client remove the weakness indicator? If the server wants to make x and W/x match, it needs to implement Etag matching that way. But clients should treat etags as opaque strings. IMHO.
> Why would a client remove the weakness indicator? Because the weakness indicator sent by Apache is nonsense. When I send a PUT request and immediately thereafter send a HEAD request, I always get a weak Etag from Apache, say W/"19e60b-20-279033c0". If I do the HEAD request some seconds later, I get the strong Etag "19e60b-20-279033c0". Neither Apache nor somebody else changed the content, it is just what I sent in the PUT request. And this makes no sense to the client. The reason is in modules/http/http_etag.c, ap_make_etag(): * If the request was made within a second of the last-modified date, * we send a weak tag instead of a strong one, since it could * be modified again later in the second, and the validation * would be incorrect. What should be the sense of this (would be nice if you could explain it to me)? - changes may happen at any time. Why are young files bad and old ones good? - are there race conditions within in Apache, so the Etag will not match the body of the response (mtime and etag are evaluated at one time, the response-body some time later)? In this case a weak Etag is just as wrong as strong one. Why should this race condition occur only within 1 second after the file has been modified? If this realy is the case, it needs debugging. - when the Etag matches the body of the response, it is completely ok to change the content on the server 0.1 microsecond later (because this will change Etag). As long as Apache (or some module) does not distinguish between "semantically significant changes" and changing some byte, there is no reason for weak Etags (see RFC 2616, 13.3.3 Weak and Strong Validators). For any caching WebDAV-client, it is essential to get the Etag of files uploaded to the server. If this is impossible, the client has to throw away the local copy and download it from the server again -- but only after waiting at least one second. Real world: As long as one uses only standard WebDAV (RFC 4918) with Apache mod_dav (I don't know about extension like versioning), or any other WebDAV-server, removing the weakness indicator is no problem at all. davfs2 does it, and I never heard of any problem that might be related to this. P.S.: Servers, that don't edit the body of a PUT, should send a strong Etag and Last-Modiefied in the PUT-response, allthough the WebDAV Working Group was not able to address this problem. It would avoid race conditions.
(In reply to comment #7) > > Why would a client remove the weakness indicator? > Because the weakness indicator sent by Apache is nonsense. But that doesn't mean that people should apply hacks to their clients. > When I send a PUT request and immediately thereafter send a HEAD request, I > always get a weak Etag from Apache, say W/"19e60b-20-279033c0". If I do the HEAD > request some seconds later, I get the strong Etag "19e60b-20-279033c0". Neither > Apache nor somebody else changed the content, it is just what I sent in the PUT > request. And this makes no sense to the client. It makes perfect sense for clients that just need a weak etag, such as for making GET in the browser conditional. > The reason is in modules/http/http_etag.c, ap_make_etag(): > > * If the request was made within a second of the last-modified date, > * we send a weak tag instead of a strong one, since it could > * be modified again later in the second, and the validation > * would be incorrect. > > What should be the sense of this (would be nice if you could explain it to me)? > > - changes may happen at any time. Why are young files bad and old ones good? As long as the timestamp of the file equals the system time, it can't be used to compute a strong etag (because the file can change again in the same interval). Once it's not the same anymore, it can be used to compute a strong etag. > - are there race conditions within in Apache, so the Etag will not match the > body of the response (mtime and etag are evaluated at one time, the > response-body some time later)? In this case a weak Etag is just as wrong as > strong one. Why should this race condition occur only within 1 second after the > file has been modified? If this realy is the case, it needs debugging. > > - when the Etag matches the body of the response, it is completely ok to change > the content on the server 0.1 microsecond later (because this will change Etag). That depends on the resolution of the system clock. > As long as Apache (or some module) does not distinguish between "semantically > significant changes" and changing some byte, there is no reason for weak Etags > (see RFC 2616, 13.3.3 Weak and Strong Validators). > > For any caching WebDAV-client, it is essential to get the Etag of files uploaded > to the server. If this is impossible, the client has to throw away the local > copy and download it from the server again -- but only after waiting at least > one second. Yes, that's a problem. But putting hacks into the clients (removing the weakness indicator) is the wrong way to handle this. > Real world: As long as one uses only standard WebDAV (RFC 4918) with Apache > mod_dav (I don't know about extension like versioning), or any other > WebDAV-server, removing the weakness indicator is no problem at all. davfs2 does > it, and I never heard of any problem that might be related to this. That's because nobody has tested with other WebDAV servers that may assign weak etags for other reasons than the one you see in Apache/moddav. > P.S.: Servers, that don't edit the body of a PUT, should send a strong Etag and > Last-Modiefied in the PUT-response, allthough the WebDAV Working Group was not > able to address this problem. It would avoid race conditions. Actually, servers should send the ETag always, no matter whether the body was changed (IMHO). See proposal in http://greenbytes.de/tech/webdav/draft-reschke-http-etag-on-write-latest.html (follow ups with respect to this on the http-wg mailing list, please).
Looks like this is the wrong place for our discussion. So I created a new bug report. (#42987 Weak Etags in Apache are useless and violate RFC 2616, 13.3.3) Please have a look at the test cases for the 'perfect sense' of apache-style weak etags in a conditional GET.
This bug is not specific to WebDAV! If-None-Match is a pure HTTP construct and as such fixing this bug should not touch mod_dav. I'll post a patch shortly.
Created attachment 21295 [details] Clean fix This patch comes from mod_dav_acl-0.1.2 and was written by Jari Urpalainen. Please consider applying and closing this bug.
RFC 2616 says "...or if "*" is given and any current entity exists for that resource, then the server MUST NOT perform the requested method." Therefore, this patch assumes that the absence of an etag implies the absence of the entity. Is this an assumption we want to make?
also see the discussion at: http://mail-archives.apache.org/mod_mbox/httpd-dev/200710.mbox/%3c470E9A9F.8020202@pearsoncmg.com%3e http://mail-archives.apache.org/mod_mbox/httpd-dev/200711.mbox/%3c1b4c87db0711190838v69dd7593l15c0ceb4e4755b01@mail.gmail.com%3e
(In reply to comment #12) > Is this an assumption we want to make? I'm not qualified to provide advice on that question. But please note that the patch can easily be modified if this assumption turns out not to be valid. So the only thing preventing this bug from being closed is making this decision.
(In reply to comment #13) > http://mail-archives.apache.org/mod_mbox/httpd-dev/200710.mbox/%3c470E9A9F.8020202@pearsoncmg.com%3e > http://mail-archives.apache.org/mod_mbox/httpd-dev/200711.mbox/%3c1b4c87db0711190838v69dd7593l15c0ceb4e4755b01@mail.gmail.com%3e I didn't read everything slowly, but isn't all this related to a different bug? I mean, the problem in #38034 is fixed in a correct way easily enough, without refactoring.
> I'm not qualified to provide advice on that question. But please note that the > patch can easily be modified if this assumption turns out not to be valid. So > the only thing preventing this bug from being closed is making this decision. The main thing preventing this bug from being closed is an actual commit to the source code. This bug has been opened for almost 2 years (16593 has been open for over 4.5 years!) and has seen 3 or 4 proposed patches. When a bug sees this many patches and no action, then there is a scaling problem somewhere in the development process. Adoption of the litmus webdav test suite would also be good to prevent regressions. I spoke to Greg Stein at ApacheCon last month about this bug and he mentioned that he had tested If-Match / If- None-Match behavior when he originally wrote mod_dav. Unfortunately, mod_dav has regressed in that regard. An automated test would have caught the regression.
(In reply to comment #14) > I'm not qualified to provide advice on that question. But please note that the > patch can easily be modified if this assumption turns out not to be valid. So > the only thing preventing this bug from being closed is making this decision. (Ok, my last reply was me venting. Here is my more productive response ;-) The patch may be easily modified to any particular state, but deciding on that state is the hard part ;-) In this case, if the assumption is not valid (which I do not believe it is), then we must decide on how we signify that a resource does not exist (i.e. is null). The email thread I pointed you to discusses that issue somewhat. Chris Darroch proposed NON_EXTANT_RESOURCE or NO_RESOURCE. Paritosh had already submitted a patch with "resource-exists" but then later agreed with Chris on using NO_RESOURCE (a trivial change to the patch). Then in the next month, after discussing this bug with Paul Querna at ApacheCon, Paritosh attempted to revive the thread and proposed another possible approach endorsed by Paul. No one replied to Paritosh's email. At that point, Paritosh and I decided not to invest more time in creating and testing yet another patch which may not make it into Apache. We're not opposed to doing so in the future, but we'd like to get our own automated testing infrastructure setup specifically for our patches to Apache (there are more to come). Testing that mod_dav_fs still works by hand for every patch (and every time a patch needs to be changed) is time consuming. Right now, we are under increasing pressure to tend more to our non-open-source-community tasks at our company. We hope to devote more time to pushing fixes for bugs such as this one in the near future. Hopefully, your and our efforts will not be in vain. btw, please vote for this bug if you haven't done so already.
The patch proposed by Simon Perreault only treats the bug in "If-None-Match: *", but the same bug is in "If-Match: *" and must be fixed too. >> Is this an assumption we want to make? >I'm not qualified to provide advice on that question. But please note that the >patch can easily be modified if this assumption turns out not to be valid. So >the only thing preventing this bug from being closed is making this decision. Evaluation of "If-Match: *" and "If-None-Match: *" depends on whether the resources does *exist*. I do not know, whether checking for the existence of an Etag is equivalent to checking for the existence of the resource. But if you want to do it this way, you must *know*. I am worrying about the idea of making a decision about making an *assumption*. >This bug is not specific to WebDAV! If-None-Match is a pure HTTP construct and >as such fixing this bug should not touch mod_dav. This is true, and it is wrong. Most applications seem not to use "If-None-Match: *" and "If-Match: *" and will therefore not be affected by this bug. But these conditionals are essential for WebDAV. So - it would be *nice* to have a clean and general solution - it is *necessary* to fix that bug for WebDAV. As it is, a WebDAV-client can either work reliable or work with Apache. These two options are exclusive. Cheers Werner
Additional remark an equivalence of "check for existence" and "check for Etag": I am not familiar with apache programming, so this is based on one assumption. - Apache modules can register their own, specialised ap_make_etag-function, overriding apaches generic ap_make_etag-function. If this assumption is true, it would be perfectly reasonable for a module, to return an etag only if the resource is cacheable, and to return NULL if the resource is not cacheable. So checking for the existance of an etag can not replace the check for existance of the resource. I think a clean, general solution should be in the line of the patch provided by Paritosh Shah. There must also be a clean solution for the potential problems considerd by Paritosh Shah and Chris Darroch. As I understand, a clean solution might possibly change some internal interface and possibly affect other modules. I fully understand that this needs serious consideration and might take some time. If it is therefore not possible to fix this bug in a clean, general way for the next release, I suggest that the next release should fix the bug for mod_dav only (so it will not affect other modules). You might use that ugly, code dublicating monster from me. As soon as a better solution is found, this mod_dav-only patch can be removed without side effects. Werner
Created attachment 21343 [details] Fix against 2.2.x Werner, can you please confirm that the attached patch against 2.2.x solves your problem? This is the version of the patch that should be backported.
Referring to comment #20: I applied the patch to the Debian/Etch-version of Apache 2.2.3 and only exchanged mod_dav.so in the installed binary version. All my tests succeeded (no errors). The tests included (with response code): LOCK If-None-Match: *, file does not exist 200 OK PUT If-None-Match: *, file does not exist 201 Created PUT If-None-Match: *, file does exist 412 Precondition Failed PUT If-Match: *, file does exist 204 No Content PUT If-Match: "af508-2c-69e15c40", etag matches existing file 204 No Content PUT If-Match: "quatsch", etag seems to not exactly match existing file 412 Precondition Failed Thanks Werner
Just to be crystal clear: Everything is now as you expect, right?
Yes! Your patch fixes bug 38034. Werner
Thanks for confirmation. I am sorry to say that it is likely that the patch missed the boat for 2.2.7, but it is highly likely that it will be part of 2.2.8 as it already has two votes for backport and only misses one: http://svn.apache.org/viewvc?view=rev&revision=609024 http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/STATUS?view=markup&pathrev=609024 But at least a accepted patch that is already in trunk is now available. Thanks for being persistent.
Just a question related to this issue - not sure if I'm 100% correct in my thinking yet... Did ap_meet_condition fail because the ETag for the non-existant file is constantly changing (mtime only ETag from http_etag.c) as there is no finfo (as a by product of it not existing)? locks on non-existent files create a .DAV/.locknull so a resource does actually exist (a lock-null resource)? is this correct? I'm happy with the present work-around but if what i'm thinking is the case then a cleaner fix in the future could be having mod_dav always providing its ETags (overriding http_etag for DAV directories) and as it knows about lock-null resources it could use the .locknull file for a constant inode-size-mtime Etag instead of just mtime, making ap_meets_condition happy? That's if my assumptions are correct.
In reply to comment #25: Lock-null resources do *not* exist. Only the name is locked to prevent other clients from creating a resource with that name. Lock-null resources have no etag and no mtime associated. GET requests on lock-null resources will fail with 404 NOT FOUND. An LOCK If-None-Match: * must fail with "423 LOCKED" (not 412 PRECONDITION FAILED). Finally: Locked-null resources are deprecated by RFC 4918 in favour of locked-empty resources, which do exist. They will probably disappear in an overhaul of mod_dav. Etags in mod_dav and mod_dav_fs should be handled separately from the Apache core. This is an open issue which cannot be solved that easy. Please see Bug report #42987 as well as the discussion thread starting at http://mail-archives.apache.org/mod_mbox/httpd-dev/200710.mbox/%3c470E9A9F.8020202@pearsoncmg.com%3e Werner
Sorry, it's me again. There seems to be a related bug in the way apache/mod_dav handles conditional PUT with header If-Unmodified-Since. It will always fail because ap_meets_conditions does not know the mtime of the resource. This bug will not show up most of the time as etag is checked first. I only noticed it, because a bug in davfs2 caused a PUT-request without If-Match-header and with If-Unmodified-Since-header. As the interface documentation of ap_meets_conditions in include/http_protocal.h says, ap_meets_conditions is only ment for GET requests. It can't work with PUT. So a future revision should either change ap_meets_conditions, as proposed by Paritosh, or mod_dav should handle conditionals all of it's own (taking into account that the requirements of WebDAV are quite different in this respect). Werner
> So a future revision should either change ap_meets_conditions, as proposed by > Paritosh, or mod_dav should handle conditionals all of it's own (taking into > account that the requirements of WebDAV are quite different in this respect). The requirements fot WebDAV are exactly the same as for plain HTTP, except for the addition of the "If" header. Or am I missing something?
In reply to comment #28: Yes, you missed the point. Apache core does not handle PUT-requests and ap_meets_conditions is designed for GET/HEAD-requests only (this is documented behaviour). This is perfectly OK for the vast majority of Web-servers (they don't need and don't want PUT). WebDAV is about authoring and PUT is essential. Why ap_meets_conditions cannot work with PUT: ap_meets_conditions compares the validators from the request with the validators from the response. This is OK for GET. With PUT-requests, the validators from the request have to be compared to the validators associated with the stored entity before the PUT-body is stored. The validators in the response will be different. It is up to the decision by Apache developers, whether they want to - change ap_meets_conditions (this will change the interface), or - leave it to modules like mod_dav to check the conditions according to their needs. Werner
What you're describing are the differences between Apache httpd and moddav, not between RFC2616 and RFC4918. PUT is part of RFC2616, so are all conditional headers (except "If"). Maybe an HTTP server implementation that does not support PUT can get away with a simpler *implementation*, but that doesn't really change the required semantics.
Hello Julian, the header ot this page says "ASF Bugzilla". I assume "A" stands for Apache, not for Anything. Werner
(In reply to comment #29) > the vast majority of Web-servers (they don't need and don't want PUT). WebDAV is If Apache just wants to keep the status quo, then yes. But PUT is showing up in REST-style applications (although disguised as a POST) and in XForms. Core Apache may eventually want to care about PUT
Fixed in 2.2.8.