HttpComponents HttpClient
  1. HttpComponents HttpClient
  2. HTTPCLIENT-533

CookiePolicy.registerCookieSpec(CookiePolicy.DEFAULT, <Some CookieSpec>); does not work as documented

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 3.0 RC3
    • Fix Version/s: 3.1 Alpha 1
    • Component/s: HttpClient
    • Labels:
      None
    • Environment:
      Operating System: Windows 2000
      Platform: PC

      Description

      I use HtmlUnit to access some quote information. Cookies shall be ignored,so I
      set the default policy to CookiePolicy.IGNORE_COOKIES. Nevertheless I get error
      messages after starting my program.

      The code reads:
      ---------------------------------------
      CookiePolicy.registerCookieSpec(CookiePolicy.DEFAULT, IgnoreCookiesSpec.class);
      final WebClient webClient = new WebClient();
      final URL url = new URL("http://de.finance.yahoo.com/q?s=CB3569.SG");
      final HtmlPage page = (HtmlPage) webClient.getPage(url);
      ----------------------------------------

      The error messages are:
      ----------------------------------------
      WARNUNG: Cookie rejected: "$Version=0; PRF=&t=CB3569.SG; $Domain=finance.yahoo.c
      om; $Path=/". Domain attribute "finance.yahoo.com" violates RFC 2109: domain mus
      t start with a dot
      30.11.2005 10:28:20 org.apache.commons.httpclient.HttpMethodBase processResponse
      Headers
      WARNUNG: Cookie rejected: "$Version=0; B=7fkoh9l1oqs4h&b=3&s=hb; $Domain=.yahoo.
      com; $Path=/". Domain attribute ".yahoo.com" violates RFC 2109: host minus domai
      n may not contain any dots
      ----------------------------------------

      1. ASF.LICENSE.NOT.GRANTED--patch.txt
        1 kB
        Ortwin Glück
      2. ASF.LICENSE.NOT.GRANTED--java0.log.0
        58 kB
        Bernhard Petri
      3. ASF.LICENSE.NOT.GRANTED--docs-patch.diff
        0.8 kB
        Ortwin Glück

        Activity

        Hide
        Oleg Kalnichevski added a comment -

        Patch checked in

        Oleg

        Show
        Oleg Kalnichevski added a comment - Patch checked in Oleg
        Hide
        Ortwin Glück added a comment -

        Documentation patch committed.

        Show
        Ortwin Glück added a comment - Documentation patch committed.
        Hide
        Oleg Kalnichevski added a comment -

        Patch away, Odi

        Oleg

        Show
        Oleg Kalnichevski added a comment - Patch away, Odi Oleg
        Hide
        Ortwin Glück added a comment -

        Created an attachment (id=17156)
        Patch for 3.0 docs

        Show
        Ortwin Glück added a comment - Created an attachment (id=17156) Patch for 3.0 docs
        Hide
        Oleg Kalnichevski added a comment -

        Works for me. See for yourself

        Test code run against local Tomcat 5.5.9:
        ==========
        HttpClient httpclient = new HttpClient();
        GetMethod httpget = new
        GetMethod("http://localhost:8080/servlets-examples/servlet/CookieExample");
        httpget.setQueryString(new NameValuePair[]

        { new NameValuePair("cookiename", ""), // invalid name new NameValuePair("cookievalue", "value")}

        );
        httpget.getParams().setCookiePolicy(CookiePolicy.IGNORE_COOKIES);
        try

        { httpclient.executeMethod(httpget); System.out.println(httpget.getStatusLine()); }

        finally

        { httpget.releaseConnection(); }

        Default cookie policy / RFC2109
        ================================
        [DEBUG] header - >> "GET
        /servlets-examples/servlet/CookieExample?cookiename=&cookievalue=value
        HTTP/1.1[\r][\n]"
        [DEBUG] header - >> "User-Agent: Jakarta Commons-HttpClient/3.0-rc4[\r][\n]"
        [DEBUG] header - >> "Host: localhost:8080[\r][\n]"
        [DEBUG] header - >> "[\r][\n]"
        [DEBUG] header - << "HTTP/1.1 200 OK[\r][\n]"
        [DEBUG] header - << "Server: Apache-Coyote/1.1[\r][\n]"
        [DEBUG] header - << "Set-Cookie: =value[\r][\n]"
        [DEBUG] header - << "Content-Type: text/html;charset=ISO-8859-1[\r][\n]"
        [DEBUG] header - << "Content-Length: 731[\r][\n]"
        [DEBUG] header - << "Date: Fri, 02 Dec 2005 13:51:40 GMT[\r][\n]"
        [WARN] HttpMethodBase - Invalid cookie header: "=value". Cookie name may not be
        blank

        Ignore cookie policy
        ================================
        [DEBUG] header - >> "GET
        /servlets-examples/servlet/CookieExample?cookiename=&cookievalue=value
        HTTP/1.1[\r][\n]"
        [DEBUG] header - >> "User-Agent: Jakarta Commons-HttpClient/3.0-rc4[\r][\n]"
        [DEBUG] header - >> "Host: localhost:8080[\r][\n]"
        [DEBUG] header - >> "[\r][\n]"
        [DEBUG] header - << "HTTP/1.1 200 OK[\r][\n]"
        [DEBUG] header - << "Server: Apache-Coyote/1.1[\r][\n]"
        [DEBUG] header - << "Set-Cookie: =value[\r][\n]"
        [DEBUG] header - << "Content-Type: text/html;charset=ISO-8859-1[\r][\n]"
        [DEBUG] header - << "Content-Length: 731[\r][\n]"
        [DEBUG] header - << "Date: Fri, 02 Dec 2005 13:51:05 GMT[\r][\n]"

        Please debug WebClient class

        Oleg

        Show
        Oleg Kalnichevski added a comment - Works for me. See for yourself Test code run against local Tomcat 5.5.9: ========== HttpClient httpclient = new HttpClient(); GetMethod httpget = new GetMethod("http://localhost:8080/servlets-examples/servlet/CookieExample"); httpget.setQueryString(new NameValuePair[] { new NameValuePair("cookiename", ""), // invalid name new NameValuePair("cookievalue", "value")} ); httpget.getParams().setCookiePolicy(CookiePolicy.IGNORE_COOKIES); try { httpclient.executeMethod(httpget); System.out.println(httpget.getStatusLine()); } finally { httpget.releaseConnection(); } Default cookie policy / RFC2109 ================================ [DEBUG] header - >> "GET /servlets-examples/servlet/CookieExample?cookiename=&cookievalue=value HTTP/1.1 [\r] [\n] " [DEBUG] header - >> "User-Agent: Jakarta Commons-HttpClient/3.0-rc4 [\r] [\n] " [DEBUG] header - >> "Host: localhost:8080 [\r] [\n] " [DEBUG] header - >> " [\r] [\n] " [DEBUG] header - << "HTTP/1.1 200 OK [\r] [\n] " [DEBUG] header - << "Server: Apache-Coyote/1.1 [\r] [\n] " [DEBUG] header - << "Set-Cookie: =value [\r] [\n] " [DEBUG] header - << "Content-Type: text/html;charset=ISO-8859-1 [\r] [\n] " [DEBUG] header - << "Content-Length: 731 [\r] [\n] " [DEBUG] header - << "Date: Fri, 02 Dec 2005 13:51:40 GMT [\r] [\n] " [WARN] HttpMethodBase - Invalid cookie header: "=value". Cookie name may not be blank Ignore cookie policy ================================ [DEBUG] header - >> "GET /servlets-examples/servlet/CookieExample?cookiename=&cookievalue=value HTTP/1.1 [\r] [\n] " [DEBUG] header - >> "User-Agent: Jakarta Commons-HttpClient/3.0-rc4 [\r] [\n] " [DEBUG] header - >> "Host: localhost:8080 [\r] [\n] " [DEBUG] header - >> " [\r] [\n] " [DEBUG] header - << "HTTP/1.1 200 OK [\r] [\n] " [DEBUG] header - << "Server: Apache-Coyote/1.1 [\r] [\n] " [DEBUG] header - << "Set-Cookie: =value [\r] [\n] " [DEBUG] header - << "Content-Type: text/html;charset=ISO-8859-1 [\r] [\n] " [DEBUG] header - << "Content-Length: 731 [\r] [\n] " [DEBUG] header - << "Date: Fri, 02 Dec 2005 13:51:05 GMT [\r] [\n] " Please debug WebClient class Oleg
        Hide
        Bernhard Petri added a comment -

        Created an attachment (id=17121)
        Context and wire log

        I have attached the context and wire log you wanted.

        Show
        Bernhard Petri added a comment - Created an attachment (id=17121) Context and wire log I have attached the context and wire log you wanted.
        Hide
        Ortwin Glück added a comment -

        That's especially weird, because for this there exists a test case.

        Show
        Ortwin Glück added a comment - That's especially weird, because for this there exists a test case.
        Hide
        Oleg Kalnichevski added a comment -

        Please produce a wire/context log of the session

        Oleg

        Show
        Oleg Kalnichevski added a comment - Please produce a wire/context log of the session Oleg
        Hide
        Bernhard Petri added a comment -

        After reading your discussion I have tried the following to implement a
        IGNORE_COOKIES policy:
        -------------------------------------
        HttpMethod method = new GetMethod();
        method.getParams().setCookiePolicy(CookiePolicy.IGNORE_COOKIES);
        final WebClient webClient = new WebClient();
        final URL url = new URL("http://de.finance.yahoo.com/q?s=CB3569.SG");
        final HtmlPage page = (HtmlPage) webClient.getPage(url);
        -------------------------------------
        But this also doesn't work, i.e. I get the same error messages as before.

        Show
        Bernhard Petri added a comment - After reading your discussion I have tried the following to implement a IGNORE_COOKIES policy: ------------------------------------- HttpMethod method = new GetMethod(); method.getParams().setCookiePolicy(CookiePolicy.IGNORE_COOKIES); final WebClient webClient = new WebClient(); final URL url = new URL("http://de.finance.yahoo.com/q?s=CB3569.SG"); final HtmlPage page = (HtmlPage) webClient.getPage(url); ------------------------------------- But this also doesn't work, i.e. I get the same error messages as before.
        Hide
        Ortwin Glück added a comment -

        Yeah I agree that for the attached patch it is too late.

        Show
        Ortwin Glück added a comment - Yeah I agree that for the attached patch it is too late.
        Hide
        Oleg Kalnichevski added a comment -

        Certainly for the 3.0 release

        Oleg

        Show
        Oleg Kalnichevski added a comment - Certainly for the 3.0 release Oleg
        Hide
        Ortwin Glück added a comment -

        So you vote for changing the docs?

        Show
        Ortwin Glück added a comment - So you vote for changing the docs?
        Hide
        Oleg Kalnichevski added a comment -

        (In reply to comment #6)
        > I somehow agree. But the knowledge about the default spec is now in two places:
        > 1. CookieSpec class
        > 2. DefaultHttpParamsFactory class
        >
        > Why?

        Well, because HttpClient 2.0 and 3.0 API is a bloody mess. This said, my general
        recommendation to avoid problems with HttpClient configuration is very simple:
        if one needs specific behaviour, she should explicitly set the respective HTTP
        parameter, otherwise HttpClient will pick a default at its discretion

        Oleg

        Show
        Oleg Kalnichevski added a comment - (In reply to comment #6) > I somehow agree. But the knowledge about the default spec is now in two places: > 1. CookieSpec class > 2. DefaultHttpParamsFactory class > > Why? Well, because HttpClient 2.0 and 3.0 API is a bloody mess. This said, my general recommendation to avoid problems with HttpClient configuration is very simple: if one needs specific behaviour, she should explicitly set the respective HTTP parameter, otherwise HttpClient will pick a default at its discretion Oleg
        Hide
        Ortwin Glück added a comment -

        I somehow agree. But the knowledge about the default spec is now in two places:
        1. CookieSpec class
        2. DefaultHttpParamsFactory class

        Why?

        Show
        Ortwin Glück added a comment - I somehow agree. But the knowledge about the default spec is now in two places: 1. CookieSpec class 2. DefaultHttpParamsFactory class Why?
        Hide
        Oleg Kalnichevski added a comment -

        In my opinion HttpClient works as advertised. The default policy kicks in only
        if no policy has been explicitly defined. Per default rfc2109 is set by the
        DefaultHttpParamsFactory. If things are not clear I would very much rather
        change the docs

        Oleg

        Show
        Oleg Kalnichevski added a comment - In my opinion HttpClient works as advertised. The default policy kicks in only if no policy has been explicitly defined. Per default rfc2109 is set by the DefaultHttpParamsFactory. If things are not clear I would very much rather change the docs Oleg
        Hide
        Ortwin Glück added a comment -

        Created an attachment (id=17085)
        possible patch

        Okay, our docs really state
        CookiePolicy.registerCookieSpec(CookiePolicy.DEFAULT, <Some CookieSpec>);
        but the preferred way is:
        method.getParams().setCookiePolicy(CookiePolicy.RFC_2109);

        And there actually might be a problem, when using the first variant. I see that
        DefaultHttpParamsFactory sets RFC_2109 hard coded instead of DEFAULT. Why it
        does that is obscure to me. The attached patch should take care of it. What do
        you think, Oleg?

        Show
        Ortwin Glück added a comment - Created an attachment (id=17085) possible patch Okay, our docs really state CookiePolicy.registerCookieSpec(CookiePolicy.DEFAULT, <Some CookieSpec>); but the preferred way is: method.getParams().setCookiePolicy(CookiePolicy.RFC_2109); And there actually might be a problem, when using the first variant. I see that DefaultHttpParamsFactory sets RFC_2109 hard coded instead of DEFAULT. Why it does that is obscure to me. The attached patch should take care of it. What do you think, Oleg?
        Hide
        Oleg Kalnichevski added a comment -

        Per default HttpClient is configured to use the 'rfc2109' cookie policy. Make
        sure you change it to 'default'

        Oleg

        Show
        Oleg Kalnichevski added a comment - Per default HttpClient is configured to use the 'rfc2109' cookie policy. Make sure you change it to 'default' Oleg
        Hide
        Bernhard Petri added a comment -

        (In reply to comment #1)
        > This is not a bug, but your code is wrong. Please read
        > http://jakarta.apache.org/commons/httpclient/cookies.html on how to use a cookie
        > spec. You may subscribe to the users mailing list for help. This is a bug
        > reporting tool and not a support ticketing system.
        I have read the above reference and also the API documentation carefully before
        opening a bug report. I have read both documents again, but either the documents
        are wrong or there is a bug. So please check ist again.

        Show
        Bernhard Petri added a comment - (In reply to comment #1) > This is not a bug, but your code is wrong. Please read > http://jakarta.apache.org/commons/httpclient/cookies.html on how to use a cookie > spec. You may subscribe to the users mailing list for help. This is a bug > reporting tool and not a support ticketing system. I have read the above reference and also the API documentation carefully before opening a bug report. I have read both documents again, but either the documents are wrong or there is a bug. So please check ist again.
        Hide
        Ortwin Glück added a comment -

        This is not a bug, but your code is wrong. Please read
        http://jakarta.apache.org/commons/httpclient/cookies.html on how to use a cookie
        spec. You may subscribe to the users mailing list for help. This is a bug
        reporting tool and not a support ticketing system.

        Show
        Ortwin Glück added a comment - This is not a bug, but your code is wrong. Please read http://jakarta.apache.org/commons/httpclient/cookies.html on how to use a cookie spec. You may subscribe to the users mailing list for help. This is a bug reporting tool and not a support ticketing system.

          People

          • Assignee:
            Unassigned
            Reporter:
            Bernhard Petri
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development