Bug 44679 - Cookies are treated differently between 6.0.16 and 6.0.14
Summary: Cookies are treated differently between 6.0.16 and 6.0.14
Status: RESOLVED FIXED
Alias: None
Product: Tomcat 6
Classification: Unclassified
Component: Catalina (show other bugs)
Version: 6.0.16
Hardware: PC Windows XP
: P2 regression (vote)
Target Milestone: default
Assignee: Tomcat Developers Mailing List
URL:
Keywords:
: 44685 44871 46177 46603 47429 49252 (view as bug list)
Depends on:
Blocks:
 
Reported: 2008-03-26 08:40 UTC by Richard Hensley
Modified: 2014-02-17 13:42 UTC (History)
10 users (show)



Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Richard Hensley 2008-03-26 08:40:15 UTC
I have a cookie that has a value of 

42200@5f3074c56c674ef2b092a8722d7c2dfe@c2af4f71ba69af352dfd8dba77e4c355

I use the following code to find it.

            Cookie[] cookies = httpRequest.getCookies();
            if (cookies != null)
            {
                for (Cookie cookie : cookies)
                {
                    if (KEY.equals(cookie.getName()))
                    {
                        cookieValue = cookie.getValue();
                        break;
                    }
                }
            }

in 6.0.14, the variable cookieValue has the value of the cookie at the end of the loop.

in 6.0.16, the variable cookieValue has the value of 42200 at the end of the loop.

This is preventing an upgrade to 6.0.16
Comment 1 Filip Hanik 2008-03-26 15:43:09 UTC
THe @ is not valid for a v0 cookie, 
In 6.0.17, Tomcat will automatically switch your cookie to v1 when you have an invalid value, hence making invalid cookies work.

So it's fixed, and will be placed in the next release

A workaround for you today, is to do
cookie.setVersion(1)
Comment 2 Richard Hensley 2008-03-26 15:45:53 UTC
Thanks for the quick reply and workaround. I will use it.
Comment 3 Filip Hanik 2008-03-26 20:04:26 UTC
*** Bug 44685 has been marked as a duplicate of this bug. ***
Comment 4 Filip Hanik 2008-03-28 19:12:39 UTC
*** Bug 44705 has been marked as a duplicate of this bug. ***
Comment 5 David Lewis 2008-03-29 22:13:51 UTC
Bug 44705 has been listed as a duplicate of this bug - but I am unsure that this is the case ... 

According the the specification for version 0 (or Netscape) cookies (reference: http://wp.netscape.com/newsref/std/cookie_spec.html) - the NAME=VALUE portion of a cookie "is a sequence of characters excluding semi-colon, comma and white space". The issue that I have been experiencing with Tomcat 5.5.26 if the cookie NAME name contains a colon (":"), the portion of the name that prefixes the colon remains, but the value for the cookie is an empty string.

Simply put - colons are valid characters for version 0 and version 1 (RFC 2109/RFC 2965) cookies.

Below is a reproducible example of the problem. It is important to note that the cookies are initially sent to the Web browser as version 1 cookies, however when the browser send the cookies back to Tomcat when a request is made, they only contain the NAME-VALUE component. Essentially, Tomcat has no way to inherently "know" what version of cookie has been sent.

Tomcat 5.5.25 and earlier:
--------------------------
HTTP request contains header:
Cookie: JSESSIONID=87C7E668C6C15E2556C0977F6EAA9F4C; NFIS:profile=lang=en;
DACS:NFIS:NRCAN:dlewis=usjp1tigSqakP8BnWv

* When the HttpServletRequest.getCookies() is called, a Cookie[] is returned:
Cookie: name="JSESSIONID", value="J87C7E668C6C15E2556C0977F6EAA9F4C"
Cookie: name="NFIS:profile", value="lang:en"
Cookie: name="DACS:NFIS:NRCAN:dlewis", value="usjp1tigSqakP8BnWv"

Tomcat 5.5.26:
--------------
HTTP request contains header:
Cookie: JSESSIONID=87C7E668C6C15E2556C0977F6EAA9F4C; NFIS:profile=lang=en;
DACS:NFIS:NRCAN:dlewis=usjp1tigSqakP8BnWv

* When the HttpServletRequest.getCookies() is called, a Cookie[] is returned:
Cookie: name="JSESSIONID", value="J87C7E668C6C15E2556C0977F6EAA9F4C"
Cookie: name="NFIS", value=""
Cookie: name="DACS", value=""

This issue "breaks" many of the cookies that we use with our applications. The
only solution (for now) is to use Tomcat 5.5.25.

--- When is the next Tomcat 5.5.x slated for release?


Comment 6 Mark Thomas 2008-03-30 13:10:25 UTC
(In reply to comment #5)
> Simply put - colons are valid characters for version 0 and version 1 (RFC
> 2109/RFC 2965) cookies.

This caught me out recently. Colons might not be explicitly mentioned in the cookie specs but the http spec does not allow them (nor a number of other characters) unless they are quoted - which Tomcat now does for cookies it issues and requires for cookies it receives.
Comment 7 Filip Hanik 2008-03-31 07:28:42 UTC
servlet spec says

public void setValue(String newValue)

    Assigns a new value to a cookie after the cookie is created. If you use a binary value, you may want to use BASE64 encoding.

    With Version 0 cookies, values should not contain white space, brackets, parentheses, equals signs, commas, double quotes, slashes, question marks, at signs, colons, and semicolons. Empty values may not behave the same way on all browsers. 
Comment 8 David Lewis 2008-04-01 12:06:32 UTC
OK ... thanks!

I see that at the bottom of section 2.2 of RFC 2616 (HTTP/1.1)

   Many HTTP/1.1 header field values consist of words separated by LWS
   or special characters. These special characters MUST be in a quoted
   string to be used within a parameter value (as defined in section
   3.6).

       token          = 1*<any CHAR except CTLs or separators>
       separators     = "(" | ")" | "<" | ">" | "@"
                      | "," | ";" | ":" | "\" | <">
                      | "/" | "[" | "]" | "?" | "="
                      | "{" | "}" | SP | HT

http://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2.2
Comment 9 Filip Hanik 2008-04-24 14:33:12 UTC
*** Bug 44871 has been marked as a duplicate of this bug. ***
Comment 10 Mark Thomas 2008-05-02 12:55:33 UTC
*** Bug 44927 has been marked as a duplicate of this bug. ***
Comment 11 Tim McCune 2008-05-02 14:30:58 UTC
I just tried this with the latest code from the trunk in subversion, and it still fails.
Comment 12 Tim McCune 2008-05-02 14:33:09 UTC
Sorry, I meant to add that comment to bug 44927.
Comment 13 Mark Thomas 2008-06-12 05:24:44 UTC
*** Bug 45191 has been marked as a duplicate of this bug. ***
Comment 14 Philip S 2008-06-12 09:00:11 UTC
Users shouldnt have to (effectively) format a Cookies output themselves (by needing to add quotes to a Cookies value)...
Comment 15 Filip Hanik 2008-06-12 15:48:51 UTC
(In reply to comment #14)
> Users shouldnt have to (effectively) format a Cookies output themselves (by
> needing to add quotes to a Cookies value)...
> 

you don't, just call setVersion(1)

Comment 16 Mark Thomas 2008-07-31 11:59:58 UTC
*** Bug 44705 has been marked as a duplicate of this bug. ***
Comment 17 Mark Thomas 2008-07-31 13:09:11 UTC
*** Bug 44705 has been marked as a duplicate of this bug. ***
Comment 18 Mark Thomas 2008-08-03 05:56:18 UTC
*** Bug 44705 has been marked as a duplicate of this bug. ***
Comment 19 David Lewis 2008-10-09 17:07:29 UTC
Can anyone confirm that this fix is included with the recent Tomcat 5.5.27 release?
Comment 20 Tristan Tarrant 2008-10-10 08:10:04 UTC
> > Users shouldnt have to (effectively) format a Cookies output themselves (by
> > needing to add quotes to a Cookies value)...
> > 
> 
> you don't, just call setVersion(1)

I am using 6.0.18 and I believe the implementation is incorrect.

I have a token which I am BASE64 encoding and adding to the response, but Tomcat says:


java.lang.IllegalArgumentException: Control character in cookie value, consider BASE64 encoding your value

BASE64 pads strings with = characters, which according to the above definition *are* control characters, so the recommendation is wrong.

I am setting version 1 on the cookie.

Here is my code:

Cookie cookie = new Cookie(COOKIENAME,new BASE64Encoder().encode(token.getBytes()));
cookie.setVersion(1);

response.addCookie(cookie);
Comment 21 Filip Hanik 2008-10-10 09:19:40 UTC
what value does
new BASE64Encoder().encode(token.getBytes()))

return?

Comment 22 Tim Funk 2008-10-10 09:25:07 UTC
simple wrappers to strip = and \n (tweak to accurately use your own BASE64Encoder class)

    public static String decodeBase64(String s) {
        switch(s.length()%4) {
            case 3:
                s+= "==="; break;
            case 2:
                s+= "=="; break;
            case 1:
                s+= "="; break;
            default:
        }
        return new BASE64Encoder().decode(s.getBytes())););
    }

    public static String encodeBase64(String s) {
        String encoded = BASE64Encoder().encode(s.getBytes())));
        return encoded.replaceAll("[\n=]", "");
    }
Comment 23 Tristan Tarrant 2008-10-12 23:57:00 UTC
(In reply to comment #22)
> simple wrappers to strip = and \n (tweak to accurately use your own
> BASE64Encoder class)

WHAT ? I'm sorry but that is not an acceptable solution. What if the cookie is read by applications other than mine ? I have to tell them to strip and repad the token ? 

Shouldn't a version 1 cookie be quoted, therefore protecting the "=" and therefore resulting in a valid cookie ?

Otherwise the exception thrown by the following:

java.lang.IllegalArgumentException: Control character in cookie value, consider BASE64 encoding your value
        at org.apache.tomcat.util.http.ServerCookie.maybeQuote2(ServerCookie.java:365)
        at org.apache.tomcat.util.http.ServerCookie.maybeQuote2(ServerCookie.java:358)
        at org.apache.tomcat.util.http.ServerCookie.appendCookieValue(ServerCookie.java:268)


is incorrect and confusing and should be changed. Should I file a different bug for this ?

Comment 24 Mark Thomas 2008-10-13 00:11:41 UTC
To repeat Filip's question, what value does

new BASE64Encoder().encode(token.getBytes()))

return?
Comment 25 Tristan Tarrant 2008-10-13 05:09:06 UTC
(In reply to comment #24)
> To repeat Filip's question, what value does
> 
> new BASE64Encoder().encode(token.getBytes()))
> 
> return?
> 

For example

FYeqC65bZGVSlPe4Q4iCRhXxOXZ4iN8QrOsQXhYV1bUOmipkRgc9H7cZ9DdQ66+UESCSg3feeMM=

BASE64 uses the following characters:

ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789+/

and "=" as a padding character at the end. 
My question: is the "=" (and the "+" in the middle of the above string) legal when quoted as a version 1 cookie ?

Comment 26 Mark Thomas 2008-10-13 07:41:18 UTC
That value works for me and is perfectly legal for a quoted version 1 cookie (as are all the characters you list)

I don't see anything invalid in Tomcat's behaviour. You will be better served using the users list for further assistance. It would help to provide the value that is causing the error.
Comment 27 Ivan Brusic 2008-11-19 15:35:46 UTC
Can anyone confirm that this fix is included with the recent Tomcat 5.5.27
release?
(In reply to comment #19)
> Can anyone confirm that this fix is included with the recent Tomcat 5.5.27
> release?
> 

The latest version of ServerCookie is NOT in 5.5.27, only in the 6.0.x branch.

Unfortunately for me, the workaround of using cookie.setVersion(1) is not possible since the cookie is set by a third-party.  I wish I could call Google and ask them to fix things (they create the cookie), but I do not have that option.
Comment 28 David Lewis 2008-11-19 15:39:09 UTC
(In reply to comment #27)
> The latest version of ServerCookie is NOT in 5.5.27, only in the 6.0.x branch.
> 

Thanks for this information ...
Comment 29 Peter Pichler 2009-01-23 08:34:14 UTC
1.)  HTTP/1.1 allows the usage of separator chars in cookie values

It is not true, that  the RFC2616 definition for the term "token" is restricting the allowed characters for cookies following the Netscape Cookie Definition (cookie 0; see http://web.archive.org/web/20070805052634/http://wp.netscape.com/newsref/std/cookie_spec.html)

Why:
The abstract term "token" is used for the definition of possible values for some HTTP headers defined in HTTP 1.1

e.g. 
Connection = "Connection" ":" 1#(connection-token)
connection-token  = token
(http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14)

The cookie and the Set-Cookie HTTP header is not defined in Http/1.1. 

So from the point of view of Http/1.1 Cookie and Set-Cookie are so called "extension headers" (see http://www.w3.org/Protocols/rfc2616/rfc2616-sec7.html#sec7.1)

       entity-header  = Allow                    ; Section 14.7
                      | Content-Encoding         ; Section 14.11
                      | Content-Language         ; Section 14.12
                      | Content-Length           ; Section 14.13
                      | Content-Location         ; Section 14.14
                      | Content-MD5              ; Section 14.15
                      | Content-Range            ; Section 14.16
                      | Content-Type             ; Section 14.17
                      | Expires                  ; Section 14.21
                      | Last-Modified            ; Section 14.29
                      | extension-header

       extension-header = message-header

Section 4.2 defines the syntax for the abstract term "message-header"

       message-header = field-name ":" [ field-value ]
       field-name     = token
       field-value    = *( field-content | LWS )
       field-content  = <the OCTETs making up the field-value
                        and consisting of either *TEXT or combinations
                        of token, separators, and quoted-string>

The term "TEXT" is defined in Section 2.2 (Basic Rules; http://www.w3.org/Protocols/rfc2616/rfc2616-sec2.html#sec2)

        TEXT           = <any OCTET except CTLs,
                        but including LWS>

        OCTET          = <any 8-bit sequence of data>

        CTL            = <any US-ASCII control character
                        (octets 0 - 31) and DEL (127)>

        LWS            = [CRLF] 1*( SP | HT )


*********
Conclusion: HTTP/1.1 restricts possible values for cookies. Using CRT characters is not allowed. 
!!!!! It is definitely allowed to use separator characters !!!!!!
**********

2.) Servlet Spec does not allow separator chars as value for a cookie
=====================================================================
The Servlet Specification is indifferent.

The description of the Cookie Constructor says
    
    public Cookie(java.lang.String name, java.lang.String value)

    Constructs a cookie with a specified name and value.

    The name must conform to RFC 2109. That means it can contain only ASCII 
    alphanumeric characters and cannot contain commas, semicolons, or white 
    space or begin with a $ character. The cookie's name cannot be changed 
    after creation.

    The value can be anything the server chooses to send. Its value is probably 
    of interest only to the server. The cookie's value can be changed after 
    creation with the setValue method.

    By default, cookies are created according to the Netscape cookie 
    specification. The version can be changed with the setVersion method.

(see http://java.sun.com/products/servlet/2.3/javadoc/javax/servlet/http/Cookie.html#Cookie(java.lang.String,%20java.lang.String)

You can read it? "The value can be anything the server chooses to send"

You are right, the comment of the Cookie.setValue(String) Method sounds different...

    public void setValue(java.lang.String newValue)

    Assigns a new value to a cookie after the cookie is created. If you use a 
    binary value, you may want to use BASE64 encoding.

    With Version 0 cookies, values should not contain white space, brackets, 
    parentheses, equals signs, commas, double quotes, slashes, question marks, 
    at signs, colons, and semicolons. Empty values may not behave the same way
    on all browsers.

(see http://java.sun.com/products/servlet/2.3/javadoc/javax/servlet/http/Cookie.html#setValue(java.lang.String) )

It seems that the writers of the setValue - Comment made the same mistake when interpreting HTTP/1.1. (see Top 1). But they say "should",... The proposal to use the BASE64 encoding for binary information is funny, cause a BASE64 encoded binary information can contain slash and the equal character, ... characters which should not be used for a cookie 0 value....

3.) Your Work Around: Migrating cookies to version 1 Cookies, when they are containing special characters
==================================================

This is one of the most horrible ways imaginable to handle this problem...

Read what the Servlet Spec says in the setVersion() Comment...

     public void setVersion(int v)

     Sets the version of the cookie protocol this cookie complies with. Version 
     0 complies with the original Netscape cookie specification. Version 1 
     complies with RFC 2109.

     Since RFC 2109 is still somewhat new, consider version 1 as experimental; 
     do not use it yet on production sites.


There is written "DO NOT USE IT YET (Version 1 Cookies) ON PRODUCTION SITES".... 

From my point of view the justification ("is still somewhat new") is a little bit silly (RFC 2109 has been released Feb. 1997) but the conclusion is right in every case. ("DO NOT USE ON PRODUCTION SITES")

If you look outside the servlet spec, you will find there is already a new RFC (Cookie 2, RFC 2965). READ the chapter "ABSTRACT" on page 1...

   Abstract
   ...
   The method described here differs from Netscape's Cookie
   proposal [Netscape], but it can interoperate with HTTP/1.0 user
   agents that use Netscape's method.  (See the HISTORICAL section.)

   This document reflects implementation experience with RFC 2109 and
   obsoletes it.


In plain text.... RFC 2109 is incompatible with the Netscape cookie spec. It is not possible to support both in one system... So RFC 2109 is not experimental... RFC 2109 was an experiment which failed....

4.) Conclusion

* The Migration to version 1  is definitely a bug.
See http://java.sun.com/webservices/docs/1.6/api/javax/servlet/http/Cookie.html
    …By default, cookies are created using Version 0 to ensure the best interoperability.
* The newly implemented restriction concerning possible values for cookie0 can be argued with the second comments in the Cookie.setValue() method… You may say some parts of the Java-Servlet-Spec advise against the usage of separator characters….  But these characters are definitely allowed for cookie0 values in HTTP/1.1.  In previous versions it was no problem to use e.g. a Base64 encoded binary information as cookie value. (as recommended in the Servlet spec!).  From my point of view it is still a bug, that this is not possible any more. (tomcat 6.18 fixes also some security issues… The update makes troubles in existing applications… )
- So I will reopen the bug…
(I can live with it… I implemented my own Cookie0 Parser and our software does not use the Cookie-API any more… (we are implementing a security reverse proxy… and I have to handle cookies as they are defined… an even as they are used in widespread system… e.g. WebSphere uses “:” in its session cookie; a lot of applications are using BASE64 encoded strings). 

If you need a fast Cookie0 parser I can contribute one…. 
Comment 30 Mark Thomas 2009-01-25 01:32:37 UTC
(In reply to comment #29)
> 1.)  HTTP/1.1 allows the usage of separator chars in cookie values
> 
> It is not true, that  the RFC2616 definition for the term "token" is
> restricting the allowed characters for cookies following the Netscape Cookie
> Definition (cookie 0; see
> http://web.archive.org/web/20070805052634/http://wp.netscape.com/newsref/std/cookie_spec.html)

The cookie 0 spec has a number of ambiguities (eg %XX encoding is suggested but not supported by many (all?) browsers, the '=' character is not listed as one that must be quoted if used in a name or value despite the obvious issues this would cause, etc). The Tomcat treatment of v0 cookies is consistent with the v0 spec and takes a practical view of the various ambiguities that is consistent with the operation of major browsers.

The v1 spec (RFC2109) does restrict cookie values to token or quoted string.

> 2.) Servlet Spec does not allow separator chars as value for a cookie
> =====================================================================
> The Servlet Specification is indifferent.

The servlet spec defers to the v0 and v1 specs with the additional statement that:
>     With Version 0 cookies, values should not contain white space, brackets, 
>     parentheses, equals signs, commas, double quotes, slashes, question marks, 
>     at signs, colons, and semicolons. Empty values may not behave the same way
>     on all browsers.

The should is enforced as it enables parsing to be unambiguous (see the = comment above).

> 3.) Your Work Around: Migrating cookies to version 1 Cookies, when they are
> containing special characters
> ==================================================
> 
> This is one of the most horrible ways imaginable to handle this problem...

I, and I suspect a number of other committers, strongly disagree.

> There is written "DO NOT USE IT YET (Version 1 Cookies) ON PRODUCTION
> SITES".... 
> 
> From my point of view the justification ("is still somewhat new") is a little
> bit silly (RFC 2109 has been released Feb. 1997) but the conclusion is right in
> every case. ("DO NOT USE ON PRODUCTION SITES")

That statement should have been updated several spec versions ago. v1 cookies are widely supported and are a lot better than v0 in terms of a well defined spec.

> If you look outside the servlet spec, you will find there is already a new RFC
> (Cookie 2, RFC 2965). READ the chapter "ABSTRACT" on page 1...

If you read the Tomcat source you will see that the Tomcat developers are well aware of the various cookie specs.

>    Abstract
>    ...
>    The method described here differs from Netscape's Cookie
>    proposal [Netscape], but it can interoperate with HTTP/1.0 user
>    agents that use Netscape's method.  (See the HISTORICAL section.)
> 
>    This document reflects implementation experience with RFC 2109 and
>    obsoletes it.
> 
> 
> In plain text.... RFC 2109 is incompatible with the Netscape cookie spec. It is
> not possible to support both in one system... So RFC 2109 is not
> experimental... RFC 2109 was an experiment which failed....

Last time I checked browser support for RFC 2965 was patchy. It would also help if the servlet spec referenced v2 cookies.

> 4.) Conclusion
> 
> * The Migration to version 1  is definitely a bug.

We know the auto conversion isn't spec compliant. That is why if you use strict servlet compliance, the automatic conversion doesn't happen.

The fundamental problem with cookies is a lack of a well defined specification that is correctly implemented by browsers. This situation is made worse by ambiguities in the v0 spec.

The changes made to Tomcat's cookie parsing were made to address a number of security concerns. These changes had to be made. These changes highlighted a number of browser issues and further changes were made to provide a solution that is secure and compatible with major browsers.

All the known issues are addressed in 6.0.18. There may interoperability issues with some systems as a result of Tomcat's stricter parsing. These will almost certainly need to be addressed on a case by case basis but where there is a case for adding functionality or a configuration option to Tomcat it will be considered (please create a separate bugzilla entry for any such proposed patches).

As has already been noted earlier in this bug, not all of the cookie changes have been ported to 5.5.x. I will create a new Bugzilla entry for 5.5.x to track that to make sure all the changes are ported, hopefully in time for the next 5.5.x release.
Comment 31 Peter Pichler 2009-01-26 12:06:15 UTC
1-3) 
Browser Support for cookie values containing "="
================================================
I have a lot of experience with using base64 encoded binary information in cookie values. We use it since at least three years with serval 100000 requests per hour from all brosers imaginabel. There is no browser or other HTTP component (Content-Switch, proxy,firewall,...)I know, which does not support the "=" character in cookie0 values. (if you know one except the newer versions of the tomcat servlet engine... please tell me)

Beside... cookie0 spec allows "=" mark in cookie0 value... so it would be a problem of the browser, if this char would not be supported...

An "=" character in a cookie0 value does not produce an ambiguity
=================================================================
From the beginning to the first "=" mark it is the cookie name... The value starts at the first char after the "=" mark an ends when  an semicolon (;) appears (or at the end of the line). I see no problems to determinate a distinct interpretation of a cookie0 Header, if there is an equal mark character in its value. Please correct me, if I forgot anything... 

RFC2109
=======
It is not ok to argue with RFC2109, when we are talking about version0 cookies...

As already mentioned... RFC2109 has been obsoleted by RFC 2965... So it make no sense to argue with this RFC in any case

Chapter Abstract first page of RFC2965
!!!!
This document reflects implementation experience with RFC 2109 and obsoletes it.
!!!!

I agree... the basic problem is in the servlet spec
===================================================
You are right... the basic problem is the servlet spec. They still refer the obsoleted Cookie1 RFC and ignore the actual spec RFC2965. 
(further inconsitancies should be corrected; like saying "use base64 encoding for binary information" in the first sentence" and "you should not use characters like ... slash, the equal mark,..." in the next sentence...)

I do not want to bother you... I want to help to improve the quality of the tomcat engine...

I think tomcat should support cookie0 with all its features,... (It is a horror, that a .NET and a tomcat application can not share information with a base64 encoded cookie... not because .NET is so week... but because tomcat is not able to handle cookie0 values, containing a "="). 

I will not reopen this bug again, cause I do not have new arguments. (I do not understand, why this bug has been closed.... but I do not have the time to play the reopen-close game...).

Please read again my arguments and think about reopening this bug...

Especially because you are trying to solve security issues, you should take care on backward compatibility... (when writing a new HTML-app its no problem, to URL-Encode cookie-values...)
Comment 32 Flavio Crispim 2009-01-27 03:04:35 UTC
*** Bug 46603 has been marked as a duplicate of this bug. ***
Comment 33 Peter Pichler 2009-01-27 07:12:07 UTC
(In reply to comment #30)

> 
> The cookie 0 spec has a number of ambiguities (eg %XX encoding is suggested but
> not supported by many (all?) browsers, the '=' character is not listed as one
> that must be quoted if used in a name or value despite the obvious issues this
> would cause, etc). 

?? Which browser does not support URL-encoded (%XX encoding) cookie values. I know that IE 6. and 7, Firefox and Opera accecpt URL encoded Cookie values. And I do not know any browser, which does not support it... ???

It is not written explitly, but the definition "NAME=VALUE" says, that the Name Part ends with an equal mark... so it should be clear, that it is not possible to use an equal char for the name of a cookie...

You are right... within a cookie value it is not forbidden to use an equal mark by the cookie0 spec...

> If you read the Tomcat source you will see that the Tomcat developers are well
> aware of the various cookie specs.

If you aware of the various cookie specs you should know, that cookie1 is already obsoleted... and as the servlet spec says cookie0 is still state of the art... and brings the best interoperability.

> We know the auto conversion isn't spec compliant. That is why if you use 
> strict servlet compliance, the automatic conversion doesn't happen.

What is servlet spec compliance? There is one sentence in the set cookie comment ("should not use... equal mark..., slash,..") and there are at least two sentences saying the opposite...

Because you ignored them, I will repeat them...

Cookie Constructor:
    The value can be anything the server chooses to send. Its value is probably 
    of interest only to the server....
...

(see
http://java.sun.com/products/servlet/2.3/javadoc/javax/servlet/http/Cookie.html#Cookie(java.lang.String,%20java.lang.String)

Cookie.setValue(...):
    Assigns a new value to a cookie after the cookie is created. If you use a 
    binary value, you may want to use BASE64 encoding. 

(see http://java.sun.com/products/servlet/2.3/javadoc/javax/servlet/http/Cookie.html#setValue(java.lang.String)


I think you mentioned already, I am not an english native speaker... but I am sure there is a difference between "should not" and "must not". I do not know a definition for "should" and "should not" in the servlet spec... RFC 2119 (Key words for use in RFCs to Indicate Requirement Levels) gives a definition for the term "SHOULD NOT"...

    4. SHOULD NOT   This phrase, or the phrase "NOT RECOMMENDED" mean that
       there may exist valid reasons in particular circumstances when the
       particular behavior is acceptable or even useful, but the full
       implications should be understood and the case carefully weighed
       before implementing any behavior described with this label.

I think it is ok to use this RFC also for the interpretation of the servlet spec... If there is no differend definition for "SHOULD NOT" in the servlet-spec, it is not correct to say an equal mark in a cookie value is forbidden by the servlet spec... It is definitly NOT RECOMMENDED, but it is allowed....



Comment 34 Peter Pichler 2009-01-28 05:29:45 UTC
I asked the JSR315(Servlet 3.0)-team for a clarification in the next servlet spec...

>>> Peter Pichler 28.01.09 14.14 Uhr -> jsr-315-comments@jcp.org >>>
Hello!

I want to inform you about ambiguousness in the current Servlet-Spec, which was not corrected in the published draft java-doc for Servlet 3.0.

My mail is motivated by a discussion with the tomcat developers (https://issues.apache.org/bugzilla/show_bug.cgi?id=44679#c29). 

!!! Maybe you find some time to post your point of view to this bug_report, which apache currently denies acceptance. Unofficially tomcat still has the status of a “reference implementation” of the Servlet spec… so I think it makes sense to help them improving their quality!!!

Cookie-API and Cookie0 Spec / Discussion about changes made in tomcat 6.0.16-6.0.18:
========================================
Currently there is a great discussion in the tomcat-bugzilla... (https://issues.apache.org/bugzilla/show_bug.cgi?id=44679#c29) There are different points of view, but most parties agree, that the problems are provoked by inconsistencies in the Servlet-Spec.
 
Problematic part of the servlet-spec:
> public Cookie(java.lang.String name, java.lang.String value)

>    Constructs a cookie with a specified name and value.

>    The name must conform to RFC 2109. That means it can contain only ASCII alphanumeric characters and cannot 
> contain commas, semicolons, or white space or begin with a $ character. The cookie's name cannot be changed after 
> creation.

> The value can be anything the server chooses to send. Its value is probably of interest only to the server. 
> The cookie's value can be changed after creation with the setValue method. 
> ....

>> public void setValue(java.lang.String newValue)

 >   Assigns a new value to a cookie after the cookie is created. 
 
 > With Version 0 cookies, values should not contain white space, brackets, parentheses, equals signs, commas, double 
 > quotes, slashes, question marks, at signs, colons, and semicolons. Empty values may not behave the same way on all 
 > browsers.

Constructor:
The sentence in the constructor "The value can be anything the server chooses to send" is definitely too much...

setValue:
The BASE64 charset contains the slash and the equal mark... So the spec recommends base64 for binary information in the first sentence... and says it is not recommended in the second sentence...

__________
Proposal:
======
(REM You will have to translate following text from Austrian-English to real English... )

Constructor:

Replace: the text:
"The value can be anything the server chooses to send. Its value is probably of interest only to the server.”
by
"The allowed character set for cookie values depends on the cookie version used. (see setValue(...) for details)"

Use following text to explain Cookie.setValue(...) for Version 0

"Following characters may not be used for a cookie0 value: 
* RFC2616-CTL characters (ASCII 0-31 and 127; forbidden characters for extension HTTP headers)
* Semi-colon, comma and white space (explicitly forbidden by the cookie0 draft)
If you use a binary value, you may want to use BASE64 encoding."

_________________
Two further proposals for the next versions of the servlet-spec

Cookie1:
======
Fact: Cookie1-RFC (Released: 1997) has been invalidated by Cookie2 (released 2000).

So the comment for Cookie.setVersion(...) is obsolete too and should be changed.

Replace 

“Since RFC 2109 is still somewhat new, consider version 1 as experimental; do not use it yet on production sites."
by 
"RFC 2109 (Cookie-1) has been obsoleted by RFC 2965 (Cookie-2); do not use it on production sites".

Cookie2:
======
From my point of view the cookie2-Spec looks suitable. Of course browser support is week until now, but that can change quickly. The servlet spec should definitely support RFC2965 cookies.

Cookies and HttpOnly
=================
Microsoft defined this cookie feature outside any RFC. (See http://msdn.microsoft.com/en-us/library/ms533046.aspx). It is a useful additional security feature for cookies also supported by Mozilla and Opera. Currently it is not possible to create HttpOnly Cookies using the servlet API.

The HttpOnly feature complicates the usage of malicious Client-Side-Cross-Side-Scripting within HTML application.

Security Reviews of applications may claim to use this feature… and it is more work to argue against it, than to implement the cookie handling… 

Yours sincerely


Peter Pichler
Comment 35 Remy Maucherat 2009-01-28 06:34:35 UTC
No matter what happens, the behavior of cookie processing will not change again in the 5.5 and 6.0 branches, I would veto that. Feel free to pursue that little crusade of yours for future releases.

BTW: Tomcat is not the reference implementation of anything.
Comment 36 Mark Thomas 2009-01-29 02:04:16 UTC
I have tried to keep my response as brief as possible.

v0 cookies and '='
I do not see anything in the v0 spec that indicates that '=' is not permitted in the cookie name - hence the ambiguity. I am open to revisiting this if it can be shown clearly that the v0 spec does not permit '=' in the cookie name.

v1 cookies
Regardless of one's views of the v1 spec, Tomcat has to support v1 cookies. I don't see any issues with Tomcat 6's v1 cookie handling in the text above but if I have missed something, please create a separate Bugzilla entry for it.

v2 cookies
The servlet API only supports v0 and v1 cookies. I am all for adding v2 support but this would be better handled through the Expert Group (EG).

Servlet spec issues
This are best handled through the EG.

httpOnly
This is in the 3.0 servlet spec and will be supported in Tomcat 7. There are patches proposed to backport this to 6.0.x and 5.5.x but they need to be reviewed in light of the 3.0 spec changes to try and keep things consistent.

Strict servlet compliance
See the specification section of
http://tomcat.apache.org/tomcat-6.0-doc/config/systemprops.html

%XX encoding in cookie headers
I did some testing of this when looking at the cookie parsing some time ago and couldn't get it to work. My tests could have been bad. The wording in the v0 spec for %XX encoding is such that relying on any encoding scheme is going to be risky - this is one of the issues with the v0 spec.
Comment 37 Peter Pichler 2009-01-29 10:13:12 UTC
(In reply to comment #36)
> I have tried to keep my response as brief as possible.
> 
> v0 cookies and '='
> I do not see anything in the v0 spec that indicates that '=' is not permitted
> in the cookie name - hence the ambiguity. I am open to revisiting this if it
> can be shown clearly that the v0 spec does not permit '=' in the cookie name.

It is an lack in the Netscape Draft... With good will it is possible to interpretate the draft like... "it is clear, that it is not possible to have cookie names containing an equal mark, because the equal mark is the delimiter for the cookie name"... for sure you can argue otherwise also... (Other forbidden characters are mentioned explicitly... so the cookie draft allows the equal mark for cookie names...) 

The cookie draft can not be changed any more... so we have to look how to handle it... Because it is impossible to implement a Cookie-Handling allowing "=" in the cookie name... it makes no sense trying to support it...

> v1 cookies
> Regardless of one's views of the v1 spec, Tomcat has to support v1 cookies. I
> don't see any issues with Tomcat 6's v1 cookie handling in the text above but
> if I have missed something, please create a separate Bugzilla entry for it.

You are right, cause the servlet spec requires cookie1 support.... (but they say cookie1 is experimental and should not be used on production side... because of that - and because cookie1 is obsolete since more than 8 years - it may be clever to put not to much effort in the cookie1 support.... a private meaning... My topic is the cookie0 support... the cookie1 topic came up, cause others are arguing with cookie1 when talking about cookie0)


> v2 cookies, httpOnly...
You are right... v2 Cookie-Support, HttpOnly-Support should be defined first in the  servlet-spec... 

> %XX encoding in cookie headers
> I did some testing of this when looking at the cookie parsing some time ago and
> couldn't get it to work. My tests could have been bad. The wording in the v0
> spec for %XX encoding is such that relying on any encoding scheme is going to
> be risky - this is one of the issues with the v0 spec.

This is what I do in applications with cookie values previously used base64 encoding.... (to eliminate slash and equal-mark... I use base64 and URL-Encoding after, because this produces shorter result strings) ... Until now it seems to work fine . If there are realy problems with this kind of work-around... it would be interesting for me... (but we are testing our software with serveral browsers and environments - and until now no problem has been reported)

Comment 38 Matt 2009-12-23 14:25:22 UTC
I'm having the same issue as David Lewis, where my cookies have a colon in the name and therefore aren't being parsed correctly.  I've read through all the comments and I can't see anything that directly addresses the problem.  Even if I quote the name I get the same behavior.  Is this simply not supported in any way?  I never ran into this issue before because the app was running on an older version of Tomcat 5.5, and I just moved it along with another website to Tomcat 6.0.20.  I can change my code to use a different name, but I would like to be able to continue to use the cookies that are already stored in client browsers.
Comment 39 Mark Thomas 2009-12-23 14:43:05 UTC
(In reply to comment #38)
You can always parse the cookie header yourself. Note Bugzilla isn't a support forum . If you need further assistance, please use the users list.
Comment 40 Tim McCune 2010-01-25 11:50:45 UTC
I just discovered the org.apache.tomcat.util.http.ServerCookie.ALLOW_EQUALS_IN_VALUE system property that was introduced in Tomcat 6.0.24.  Looks like Mark has seen the light. :)  I just wanted to update this issue so everyone that needs this feature is aware of it.
Comment 41 Konstantin Kolinko 2010-05-05 10:42:24 UTC
*** Bug 49252 has been marked as a duplicate of this bug. ***
Comment 42 Konstantin Kolinko 2010-05-05 12:11:28 UTC
*** Bug 47429 has been marked as a duplicate of this bug. ***
Comment 43 Konstantin Kolinko 2010-05-05 12:13:09 UTC
*** Bug 46177 has been marked as a duplicate of this bug. ***
Comment 44 Krystian Nowak 2011-09-20 18:32:13 UTC
Observing the comments here I would have to support it as being an interoperability issue between other, non Java platforms/languages.
As an example we can put here Resin as Java-PHP interop bridge:
http://bugs.caucho.com/view.php?id=4767

It boils down to supporting array cookies (multidimensional arrays in fact) in PHP as described in:
http://php.net/manual/en/language.variables.external.php#language.variables.external.cookies
http://php.net/manual/en/function.setcookie.php#example-3780
http://www.developertutorials.com/tutorials/php/articlename-050526-1149/

Of course one can put one RFC against another, but fortunately, at least on different javax.servlet.http.Cookie implementations there are possible workarounds as in:
http://java.net/jira/browse/GLASSFISH-10850?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
http://java.net/jira/browse/GLASSFISH-10945?page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel
advising to use (in some implementations e.g. from javax.servlet-api-3.0.1) system property defining legacy/backward-compatibility mode:
"org.glassfish.web.rfc2109_cookie_names_enforced" system property

So it seems that the subject still lives, is viable, important and cannot be just muted by "parse the cookie header yourself".

Cheers,
Krystian