Bug 38034 - PUT If-None-Match: "*" failures
PUT If-None-Match: "*" failures
Status: RESOLVED FIXED
Product: Apache httpd-2
Classification: Unclassified
Component: mod_dav
2.0.55
Other Windows 2000
: P2 normal with 5 votes (vote)
: ---
Assigned To: Apache HTTPD Bugs Mailing List
http://greenbytes.de/tech/webdav/rfc2...
: PatchAvailable
Depends on:
Blocks:
  Show dependency tree
 
Reported: 2005-12-24 11:12 UTC by Julian Reschke
Modified: 2014-02-17 13:55 UTC (History)
3 users (show)



Attachments
test cases for LOCK request with if-* headers (3.39 KB, text/javascript)
2007-03-02 07:58 UTC, Julian Reschke
Details
Fixes If-Match: * and If-None-Match: * bug for mod_dav (6.29 KB, patch)
2007-07-26 11:30 UTC, Werner
Details | Diff
Test results for conditional LOCK-requests (1.14 KB, text/plain)
2007-07-26 11:41 UTC, Werner
Details
Clean fix (1.08 KB, patch)
2007-12-19 12:28 UTC, Simon Perreault
Details | Diff
Fix against 2.2.x (2.88 KB, patch)
2008-01-04 02:55 UTC, Ruediger Pluem
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Reschke 2005-12-24 11:12:00 UTC
PUT to an unmapped URL (DELETEd before) with

  If-None-Match: *

currently fails with 412, but should succeed (see
<http://greenbytes.de/tech/webdav/rfc2616.html#rfc.section.14.26.p.8>).
Comment 1 Julian Reschke 2007-03-02 07:56:27 UTC
It seems that If-Match and If-None-Match in general aren't evaluated properly
when the target does not exist.

For instance:

LOCK /unmapped
If-Match: "*"

should fail with 412, but creates the lock.

Or

LOCK /unmapped
If-None-Match: "*"

should succeed, but fails with 412.
Comment 2 Julian Reschke 2007-03-02 07:58:12 UTC
Created attachment 19658 [details]
test cases for LOCK request with if-* headers
Comment 3 Werner 2007-07-25 04:09:00 UTC
Isn't there any programmer somewhere to help fix this bug.
It is a *severe* bug in mod_dav and it is open for at least four years, while
IIS does it right.
Comment 4 Werner 2007-07-26 11:30:39 UTC
Created attachment 20552 [details]
Fixes If-Match: * and If-None-Match: * bug for mod_dav

The bug is in function ap_meets_conditions() in modules/http/http_protocol.c:
it always evaluates "If-Match: *" to TRUE (is FALSE, if resource does not
exist) and "If-None-Match: *" to FALSE (is TRUE, if the resource does not
exist).
This function is called by mod_dav, function dav_validate_request(). In this
case, ap_meets_conditions() seems not able to get etag reliably (probably a bug
in mod_dav).
Fix: A new function dav_meets_conditions() is created in
modules/dav/main/util.c. It is mostly a copy of ap_meets_conditions(), but
fixes the mentioned errors. dav_validate_request() calls dav_meets_conditions()
instead of ap_meets_conditions().
ToDo: it would be better to fix this in ap_meets_conditions(). But to do this,
this functions must know, whether the resource exists, and it must be able to
reliably get the etag of the resource. But as I am not familiar with
Apache-programming, I can't do this. I even doubt that it is possible without
changes in other Apache modules besides mod_dav.
Comment 5 Werner 2007-07-26 11:41:31 UTC
Created attachment 20553 [details]
Test results for conditional LOCK-requests

HTTP-body and irrelevant headers are removed.
Clients should remove "W/" from weak Etags. Apache always creates weak Etags,
when a request is sent within less than 1 second after the last modification.
So clients that use HEAD to get the Etag immediately after PUT will be fooled
when they use this Etag some seconds later.
Comment 6 Julian Reschke 2007-07-26 11:54:03 UTC
Why would a client remove the weakness indicator?

If the server wants to make x and W/x match, it needs to implement Etag matching
that way. But clients should treat etags as opaque strings. IMHO.
Comment 7 Werner 2007-07-26 14:05:55 UTC
> Why would a client remove the weakness indicator?
Because the weakness indicator sent by Apache is nonsense.

When I send a PUT request and immediately thereafter send a HEAD request, I
always get a weak Etag from Apache, say W/"19e60b-20-279033c0". If I do the HEAD
request some seconds later, I get the strong Etag "19e60b-20-279033c0". Neither
Apache nor somebody else changed the content, it is just what I sent in the PUT
request. And this makes no sense to the client.

The reason is in modules/http/http_etag.c, ap_make_etag():

     * If the request was made within a second of the last-modified date,
     * we send a weak tag instead of a strong one, since it could
     * be modified again later in the second, and the validation
     * would be incorrect.

What should be the sense of this (would be nice if you could explain it to me)?

- changes may happen at any time. Why are young files bad and old ones good?

- are there race conditions within in Apache, so the Etag will not match the
body of the response (mtime and etag are evaluated at one time, the
response-body some time later)? In this case a weak Etag is just as wrong as
strong one. Why should this race condition occur only within 1 second after the
file has been modified? If this realy is the case, it needs debugging.

- when the Etag matches the body of the response, it is completely ok to change
the content on the server 0.1 microsecond later (because this will change Etag).

As long as Apache (or some module) does not distinguish between "semantically
significant changes" and changing some byte, there is no reason for weak Etags
(see RFC 2616, 13.3.3 Weak and Strong Validators).

For any caching WebDAV-client, it is essential to get the Etag of files uploaded
to the server. If this is impossible, the client has to throw away the local
copy and download it from the server again -- but only after waiting at least
one second.

Real world: As long as one uses only standard WebDAV (RFC 4918) with Apache
mod_dav (I don't know about extension like versioning), or any other
WebDAV-server, removing the weakness indicator is no problem at all. davfs2 does
it, and I never heard of any problem that might be related to this.

P.S.: Servers, that don't edit the body of a PUT, should send a strong Etag and
Last-Modiefied in the PUT-response, allthough the WebDAV Working Group was not
able to address this problem. It would avoid race conditions.
Comment 8 Julian Reschke 2007-07-27 01:09:18 UTC
(In reply to comment #7)
> > Why would a client remove the weakness indicator?
> Because the weakness indicator sent by Apache is nonsense.

But that doesn't mean that people should apply hacks to their clients.

> When I send a PUT request and immediately thereafter send a HEAD request, I
> always get a weak Etag from Apache, say W/"19e60b-20-279033c0". If I do the HEAD
> request some seconds later, I get the strong Etag "19e60b-20-279033c0". Neither
> Apache nor somebody else changed the content, it is just what I sent in the PUT
> request. And this makes no sense to the client.

It makes perfect sense for clients that just need a weak etag, such as for
making GET in the browser conditional.
 
> The reason is in modules/http/http_etag.c, ap_make_etag():
> 
>      * If the request was made within a second of the last-modified date,
>      * we send a weak tag instead of a strong one, since it could
>      * be modified again later in the second, and the validation
>      * would be incorrect.
> 
> What should be the sense of this (would be nice if you could explain it to me)?
> 
> - changes may happen at any time. Why are young files bad and old ones good?

As long as the timestamp of the file equals the system time, it can't be used to
compute a strong etag (because the file can change again in the same interval).
Once it's not the same anymore, it can be used to compute a strong etag.

> - are there race conditions within in Apache, so the Etag will not match the
> body of the response (mtime and etag are evaluated at one time, the
> response-body some time later)? In this case a weak Etag is just as wrong as
> strong one. Why should this race condition occur only within 1 second after the
> file has been modified? If this realy is the case, it needs debugging.
> 
> - when the Etag matches the body of the response, it is completely ok to change
> the content on the server 0.1 microsecond later (because this will change Etag).

That depends on the resolution of the system clock.

> As long as Apache (or some module) does not distinguish between "semantically
> significant changes" and changing some byte, there is no reason for weak Etags
> (see RFC 2616, 13.3.3 Weak and Strong Validators).
> 
> For any caching WebDAV-client, it is essential to get the Etag of files uploaded
> to the server. If this is impossible, the client has to throw away the local
> copy and download it from the server again -- but only after waiting at least
> one second.

Yes, that's a problem. But putting hacks into the clients (removing the weakness
indicator) is the wrong way to handle this.

> Real world: As long as one uses only standard WebDAV (RFC 4918) with Apache
> mod_dav (I don't know about extension like versioning), or any other
> WebDAV-server, removing the weakness indicator is no problem at all. davfs2 does
> it, and I never heard of any problem that might be related to this.

That's because nobody has tested with other WebDAV servers that may assign weak
etags for other reasons than the one you see in Apache/moddav.

> P.S.: Servers, that don't edit the body of a PUT, should send a strong Etag and
> Last-Modiefied in the PUT-response, allthough the WebDAV Working Group was not
> able to address this problem. It would avoid race conditions.

Actually, servers should send the ETag always, no matter whether the body was
changed (IMHO). See proposal in
http://greenbytes.de/tech/webdav/draft-reschke-http-etag-on-write-latest.html
(follow ups with respect to this on the http-wg mailing list, please). 
Comment 9 Werner 2007-07-27 05:38:27 UTC
Looks like this is the wrong place for our discussion. So I created a new bug
report. (#42987 Weak Etags in Apache are useless and violate RFC 2616, 13.3.3)
Please have a look at the test cases for the 'perfect sense' of apache-style
weak etags in a conditional GET.
Comment 10 Simon Perreault 2007-12-19 12:23:42 UTC
This bug is not specific to WebDAV! If-None-Match is a pure HTTP construct and
as such fixing this bug should not touch mod_dav.

I'll post a patch shortly.
Comment 11 Simon Perreault 2007-12-19 12:28:28 UTC
Created attachment 21295 [details]
Clean fix

This patch comes from mod_dav_acl-0.1.2 and was written by Jari Urpalainen.

Please consider applying and closing this bug.
Comment 12 Tim Olsen 2007-12-19 13:22:39 UTC
RFC 2616 says "...or if "*" is given and any current entity exists for that
resource, then the server MUST NOT perform the requested method."  Therefore,
this patch assumes that the absence of an etag implies the absence of the entity.

Is this an assumption we want to make?
Comment 14 Simon Perreault 2007-12-20 05:46:01 UTC
(In reply to comment #12)
> Is this an assumption we want to make?

I'm not qualified to provide advice on that question. But please note that the
patch can easily be modified if this assumption turns out not to be valid. So
the only thing preventing this bug from being closed is making this decision.
Comment 15 Simon Perreault 2007-12-20 05:48:59 UTC
(In reply to comment #13)
>
http://mail-archives.apache.org/mod_mbox/httpd-dev/200710.mbox/%3c470E9A9F.8020202@pearsoncmg.com%3e
>
http://mail-archives.apache.org/mod_mbox/httpd-dev/200711.mbox/%3c1b4c87db0711190838v69dd7593l15c0ceb4e4755b01@mail.gmail.com%3e

I didn't read everything slowly, but isn't all this related to a different bug?
I mean, the problem in #38034 is fixed in a correct way easily enough, without
refactoring.
Comment 16 Tim Olsen 2007-12-20 08:46:37 UTC
> I'm not qualified to provide advice on that question. But please note that the
> patch can easily be modified if this assumption turns out not to be valid. So
> the only thing preventing this bug from being closed is making this decision.

The main thing preventing this bug from being closed is an actual commit to the source code.  This bug 
has been opened for almost 2 years (16593 has been open for over 4.5 years!) and has seen 3 or 4 
proposed patches.

When a bug sees this many patches and no action, then there is a scaling problem somewhere in the 
development process.

Adoption of the litmus webdav test suite would also be good to prevent regressions.  I spoke to Greg 
Stein at ApacheCon last month about this bug and he mentioned that he had tested If-Match / If-
None-Match behavior when he originally wrote mod_dav.  Unfortunately, mod_dav has regressed in 
that regard.  An automated test would have caught the regression.
Comment 17 Tim Olsen 2007-12-20 09:13:24 UTC
(In reply to comment #14) 
> I'm not qualified to provide advice on that question. But please note that the
> patch can easily be modified if this assumption turns out not to be valid. So
> the only thing preventing this bug from being closed is making this decision.

(Ok, my last reply was me venting.  Here is my more productive response ;-)

The patch may be easily modified to any particular state, but deciding on that state is the hard part ;-)  
In this case, if the assumption is not valid (which I do not believe it is), then we must decide on how we 
signify that a resource does not exist (i.e. is null).  The email thread I pointed you to discusses that 
issue somewhat.  Chris Darroch proposed NON_EXTANT_RESOURCE or NO_RESOURCE.  Paritosh had 
already submitted a patch with "resource-exists" but then later agreed with Chris on using 
NO_RESOURCE (a trivial change to the patch).   Then in the next month, after discussing this bug with 
Paul Querna at ApacheCon, Paritosh attempted to revive the thread and proposed another possible 
approach endorsed by Paul.  No one replied to Paritosh's email.

At that point, Paritosh and I decided not to invest more time in creating and testing yet another patch 
which may not make it into Apache.   We're not opposed to doing so in the future, but we'd like to get 
our own automated testing infrastructure setup specifically for our patches to Apache (there are more 
to come).  Testing that mod_dav_fs still works by hand for every patch (and every time a patch needs to 
be changed) is time consuming.  

Right now, we are under increasing pressure to tend more to our non-open-source-community tasks 
at our company.  We hope to devote more time to pushing fixes for bugs such as this one in the near 
future.  Hopefully, your and our efforts will not be in vain.

btw, please vote for this bug if you haven't done so already.

Comment 18 Werner 2007-12-21 12:15:51 UTC
The patch proposed by Simon Perreault only treats the bug in "If-None-Match: *",
but the same bug is in "If-Match: *" and must be fixed too.

>> Is this an assumption we want to make?
>I'm not qualified to provide advice on that question. But please note that the
>patch can easily be modified if this assumption turns out not to be valid. So
>the only thing preventing this bug from being closed is making this decision.

Evaluation of "If-Match: *" and "If-None-Match: *" depends on whether the
resources does *exist*. I do not know, whether checking for the existence of an
Etag is equivalent to checking for the existence of the resource. But if you
want to do it this way, you must *know*. I am worrying about the idea of making
a decision about making an *assumption*.

>This bug is not specific to WebDAV! If-None-Match is a pure HTTP construct and
>as such fixing this bug should not touch mod_dav.

This is true, and it is wrong. Most applications seem not to use "If-None-Match:
*" and "If-Match: *" and will therefore not be affected by this bug. But these
conditionals are essential for WebDAV. So
- it would be *nice* to have a clean and general solution
- it is *necessary* to fix that bug for WebDAV.

As it is, a WebDAV-client can either work reliable or work with Apache. These
two options are exclusive.

Cheers
Werner
Comment 19 Werner 2007-12-22 01:52:00 UTC
Additional remark an equivalence of "check for existence" and "check for Etag":

I am not familiar with apache programming, so this is based on one assumption.
- Apache modules can register their own, specialised ap_make_etag-function,
overriding apaches generic ap_make_etag-function.

If this assumption is true, it would be perfectly reasonable for a module, to
return an etag only if the resource is cacheable, and to return NULL if the
resource is not cacheable. So checking for the existance of an etag can not
replace the check for existance of the resource.

I think a clean, general solution should be in the line of the patch provided by
Paritosh Shah. There must also be a clean solution for the potential problems
considerd by Paritosh Shah and Chris Darroch.

As I understand, a clean solution might possibly change some internal interface
and possibly affect other modules. I fully understand that this needs serious
consideration and might take some time.

If it is therefore not possible to fix this bug in a clean, general way for the
next release, I suggest that the next release should fix the bug for mod_dav
only (so it will not affect other modules). You might use that ugly, code
dublicating monster from me. As soon as a better solution is found, this
mod_dav-only patch can be removed without side effects.

Werner
Comment 20 Ruediger Pluem 2008-01-04 02:55:55 UTC
Created attachment 21343 [details]
Fix against 2.2.x

Werner, can you please confirm that the attached patch against 2.2.x solves
your problem? This is the version of the patch that should be backported.
Comment 21 Werner 2008-01-04 11:49:48 UTC
Referring to comment #20:
I applied the patch to the Debian/Etch-version of Apache 2.2.3 and only
exchanged mod_dav.so in the installed binary version.

All my tests succeeded (no errors).

The tests included (with response code):

LOCK If-None-Match: *, file does not exist
200 OK

PUT If-None-Match: *, file does not exist
201 Created

PUT If-None-Match: *, file does exist
412 Precondition Failed

PUT If-Match: *, file does exist
204 No Content

PUT If-Match: "af508-2c-69e15c40", etag matches existing file
204 No Content

PUT If-Match: "quatsch", etag seems to not exactly match existing file
412 Precondition Failed

Thanks
Werner
Comment 22 Ruediger Pluem 2008-01-04 12:42:04 UTC
Just to be crystal clear: Everything is now as you expect, right?
Comment 23 Werner 2008-01-04 13:45:39 UTC
Yes!

Your patch fixes bug 38034.

Werner
Comment 24 Ruediger Pluem 2008-01-04 14:03:23 UTC
Thanks for confirmation. I am sorry to say that it is likely that the patch
missed the boat for 2.2.7, but it is highly likely that it will be part of 2.2.8
as it already has two votes for backport and only misses one:

http://svn.apache.org/viewvc?view=rev&revision=609024
http://svn.apache.org/viewvc/httpd/httpd/branches/2.2.x/STATUS?view=markup&pathrev=609024

But at least a accepted patch that is already in trunk is now available.
Thanks for being persistent.
Comment 25 Michael Clark 2008-01-04 22:53:10 UTC
Just a question related to this issue - not sure if I'm 100% correct in my
thinking yet...

Did ap_meet_condition fail because the ETag for the non-existant file is
constantly changing (mtime only ETag from http_etag.c) as there is no finfo
(as a by product of it not existing)?

locks on non-existent files create a .DAV/.locknull so a resource does actually
exist (a lock-null resource)? is this correct?

I'm happy with the present work-around but if what i'm thinking is the case
then a cleaner fix in the future could be having mod_dav always
providing its ETags (overriding http_etag for DAV directories) and as it knows
about lock-null resources it could use the .locknull file for a constant
inode-size-mtime Etag instead of just mtime, making ap_meets_condition happy?

That's if my assumptions are correct.
Comment 26 Werner 2008-01-05 02:12:12 UTC
In reply to comment #25:

Lock-null resources do *not* exist. Only the name is locked to prevent other
clients from creating a resource with that name.

Lock-null resources have no etag and no mtime associated.

GET requests on lock-null resources will fail with 404 NOT FOUND.

An LOCK If-None-Match: * must fail with "423 LOCKED" (not 412 PRECONDITION FAILED).

Finally:
Locked-null resources are deprecated by RFC 4918 in favour of locked-empty
resources, which do exist. They will probably disappear in an overhaul of mod_dav.

Etags in mod_dav and mod_dav_fs should be handled separately from the Apache
core. This is an open issue which cannot be solved that easy. Please see
Bug report #42987 as well as the discussion thread starting at
http://mail-archives.apache.org/mod_mbox/httpd-dev/200710.mbox/%3c470E9A9F.8020202@pearsoncmg.com%3e

Werner
Comment 27 Werner 2008-01-05 08:41:26 UTC
Sorry, it's me again.

There seems to be a related bug in the way apache/mod_dav handles conditional
PUT with header If-Unmodified-Since. It will always fail because
ap_meets_conditions does not know the mtime of the resource. This bug will not
show up most of the time as etag is checked first. I only noticed it, because a
bug in davfs2 caused a PUT-request without If-Match-header and with
If-Unmodified-Since-header.

As the interface documentation of ap_meets_conditions in include/http_protocal.h
says, ap_meets_conditions is only ment for GET requests. It can't work with PUT.
So a future revision should either change ap_meets_conditions, as proposed by
Paritosh, or mod_dav should handle conditionals all of it's own (taking into
account that the requirements of WebDAV are quite different in this respect).

Werner
Comment 28 Julian Reschke 2008-01-05 16:34:56 UTC
> So a future revision should either change ap_meets_conditions, as proposed by
> Paritosh, or mod_dav should handle conditionals all of it's own (taking into
> account that the requirements of WebDAV are quite different in this respect).

The requirements fot WebDAV are exactly the same as for plain HTTP, except for
the addition of the "If" header.

Or am I missing something?
Comment 29 Werner 2008-01-06 03:23:15 UTC
In reply to comment #28:

Yes, you missed the point.

Apache core does not handle PUT-requests and ap_meets_conditions is designed for
GET/HEAD-requests only (this is documented behaviour). This is perfectly OK for
the vast majority of Web-servers (they don't need and don't want PUT). WebDAV is
about authoring and PUT is essential.

Why ap_meets_conditions cannot work with PUT:
ap_meets_conditions compares the validators from the request with the validators
from the response. This is OK for GET.
With PUT-requests, the validators from the request have to be compared to the
validators associated with the stored entity before the PUT-body is stored. The
validators in the response will be different.

It is up to the decision by Apache developers, whether they want to
- change ap_meets_conditions (this will change the interface), or
- leave it to modules like mod_dav to check the conditions according to their needs.

Werner
Comment 30 Julian Reschke 2008-01-06 05:25:05 UTC
What you're describing are the differences between Apache httpd and moddav, not
between RFC2616 and RFC4918.

PUT is part of RFC2616, so are all conditional headers (except "If"). Maybe an
HTTP server implementation that does not support PUT can get away with a simpler
*implementation*, but that doesn't really change the required semantics.
Comment 31 Werner 2008-01-06 07:36:58 UTC
Hello Julian,

the header ot this page says "ASF Bugzilla". I assume "A" stands for Apache, not
for Anything.

Werner
Comment 32 Tim Olsen 2008-01-06 15:24:42 UTC
(In reply to comment #29)
> the vast majority of Web-servers (they don't need and don't want PUT). WebDAV is

If Apache just wants to keep the status quo, then yes.  But PUT is showing up in REST-style applications 
(although disguised as a POST) and in XForms.  Core Apache may eventually want to care about PUT 

Comment 33 Ruediger Pluem 2008-01-19 11:48:09 UTC
Fixed in 2.2.8.