CouchDB
  1. CouchDB
  2. COUCHDB-1175

Improve content type negotiation for couchdb JSON responses

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Blocker Blocker
    • Resolution: Won't Fix
    • Affects Version/s: 1.0.2
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Skill Level:
      New Contributors Level (Easy)

      Description

      Currently we ignore qvalues when negotiation between 'application/json' and 'text/plain' when returning JSON responses.

      Specifically, we test directly for 'application/json' or 'text/plain' in the Accept header. Different branches have different bugs, though. Trunk returns 'application/json' if 'application/json' is present at all, even if it's less preferred than 'text/plain' when qvalues are accounted for.

      We should follow the standard.

        Issue Links

          Activity

          Hide
          Benoit Chesneau added a comment - - edited

          that's only part of the problem.

          1. Do we want to accept text/plain where we send json . imo we should rather send 406. Wich is more restful. The problem is that it introduce some incompatibilities with previous releases.. The same for html/text plain, if this is the content we want to send then we shouldn't accept to return other content types . All of this could alos exists in the way we manage content types in shows and other functions.

          2. Auth. Since auth redirect is only useful for html content, we should better redirect only on html content and return 401 instead. This part is fixed in trunk.

          About this function, do we want that upstream or only couch ?

          Show
          Benoit Chesneau added a comment - - edited that's only part of the problem. 1. Do we want to accept text/plain where we send json . imo we should rather send 406. Wich is more restful. The problem is that it introduce some incompatibilities with previous releases.. The same for html/text plain, if this is the content we want to send then we shouldn't accept to return other content types . All of this could alos exists in the way we manage content types in shows and other functions. 2. Auth. Since auth redirect is only useful for html content, we should better redirect only on html content and return 401 instead. This part is fixed in trunk. About this function, do we want that upstream or only couch ?
          Hide
          Robert Newson added a comment -

          Precedence order has been inverted, application/json is returned if acceptable even if text/plain is also acceptable.

          Show
          Robert Newson added a comment - Precedence order has been inverted, application/json is returned if acceptable even if text/plain is also acceptable.
          Hide
          Marcello Nuccio added a comment -

          From Robert Newson response on the ML:

          The current logic for CouchDB 1.1.0 is this:

          1) If the client accepts "application/json" then respond with 401 and
          content-type "application/json" (i.e, a normal HTTP/REST response.
          2) if the client accepts "text/html" then respond with a 302 to the
          authentication_redirect url.

          This is from couch_httpd:error_headers/4.

          The problem of this logic is that it does not work with browsers like Firefox4, because they accept "/" which includes "application/json", so the "text/html" case is never triggered.

          IMHO the solution is to take "q" parameter into account, and give precedence to "text/html" if it has an higher value of "q".

          e.g., Firefox4 uses the following Accept header:

          text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8

          Here "text/html" has "q=1.0" (the default), so it should have precedence over "application/json" which has "q=0.8" (from "/").

          This should solve the "browser use-case", since browsers should always give to "text/html" higher values of "q".

          Show
          Marcello Nuccio added a comment - From Robert Newson response on the ML: The current logic for CouchDB 1.1.0 is this: 1) If the client accepts "application/json" then respond with 401 and content-type "application/json" (i.e, a normal HTTP/REST response. 2) if the client accepts "text/html" then respond with a 302 to the authentication_redirect url. This is from couch_httpd:error_headers/4. The problem of this logic is that it does not work with browsers like Firefox4, because they accept " / " which includes "application/json", so the "text/html" case is never triggered. IMHO the solution is to take "q" parameter into account, and give precedence to "text/html" if it has an higher value of "q". e.g., Firefox4 uses the following Accept header: text/html,application/xhtml+xml,application/xml;q=0.9, / ;q=0.8 Here "text/html" has "q=1.0" (the default), so it should have precedence over "application/json" which has "q=0.8" (from " / "). This should solve the "browser use-case", since browsers should always give to "text/html" higher values of "q".
          Hide
          Robert Newson added a comment -

          "The problem of this logic is that it does not work with browsers like Firefox4,"

          As I pointed out in the mailing list, it does work in Firefox 4 because I use that browser myself. Futon does not give authentication popups for me under the scenario described.

          Show
          Robert Newson added a comment - "The problem of this logic is that it does not work with browsers like Firefox4," As I pointed out in the mailing list, it does work in Firefox 4 because I use that browser myself. Futon does not give authentication popups for me under the scenario described.
          Hide
          Marcello Nuccio added a comment -

          But the problem is not with Futon.

          The problem is to DIRECTLY access the database from a browser.

          You should be redirected to a login page (CouchDB-1.0 behaviour), but you only get a "Unauthorized" error.

          I repeat: the problem is not with Futon.

          Show
          Marcello Nuccio added a comment - But the problem is not with Futon. The problem is to DIRECTLY access the database from a browser. You should be redirected to a login page (CouchDB-1.0 behaviour), but you only get a "Unauthorized" error. I repeat: the problem is not with Futon.
          Hide
          Robert Newson added a comment -

          Reproduced locally, thanks for your patience.

          Show
          Robert Newson added a comment - Reproduced locally, thanks for your patience.
          Hide
          Marcello Nuccio added a comment -

          Sorry if I gave the impression of being harsh. I only wanted to be very clear, because I am always concerned for my bad English.

          The change was deliberate but has unintended side-effects. I'm
          currently +1 on reverting this change in 1.1.1, but I'd like to
          finally nail the exact semantics for this content-type negotiation
          once and for all.

          As I said in my first comment, I think the only problem here is that we are ignoring the value of "q". This is wrong since it is not standard compliant.

          Unfortunately I have still to learn Erlang so I am not sure I can propose a patch for this...

          Show
          Marcello Nuccio added a comment - Sorry if I gave the impression of being harsh. I only wanted to be very clear, because I am always concerned for my bad English. The change was deliberate but has unintended side-effects. I'm currently +1 on reverting this change in 1.1.1, but I'd like to finally nail the exact semantics for this content-type negotiation once and for all. As I said in my first comment, I think the only problem here is that we are ignoring the value of "q". This is wrong since it is not standard compliant. Unfortunately I have still to learn Erlang so I am not sure I can propose a patch for this...
          Hide
          Robert Newson added a comment -

          I understand completely, I didn't find any comment harsh.

          I should note that I created COUCHDB-1175 with the specific intention of using q-values for this. Currently we don't use them. Before 1.1 we tested for "text/html" first and "application/json" second. Whichever you accepted first would be used. The order was inverted in 1.1.

          RFC 2616 does not require us to return the highest valued type (Section 14.1 says we SHOULD, not MUST, use the ordering), so neither 1.0 or 1.1 is technically 'not standard compliant'. Moreover, many browsers add '/' to the end of their accept list, which makes proper negotiation impossible. For example, given;

          "Accept: text/html, /"

          Both 'text/html' and 'application/json' have a q-value of 1, they are exactly equally preferred. What to do?

          Firefox sends "Accept: text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8", it's true, which means we'll send text/html, not application/json, but this doesn't work for all browsers.

          Show
          Robert Newson added a comment - I understand completely, I didn't find any comment harsh. I should note that I created COUCHDB-1175 with the specific intention of using q-values for this. Currently we don't use them. Before 1.1 we tested for "text/html" first and "application/json" second. Whichever you accepted first would be used. The order was inverted in 1.1. RFC 2616 does not require us to return the highest valued type (Section 14.1 says we SHOULD, not MUST, use the ordering), so neither 1.0 or 1.1 is technically 'not standard compliant'. Moreover, many browsers add ' / ' to the end of their accept list, which makes proper negotiation impossible. For example, given; "Accept: text/html, / " Both 'text/html' and 'application/json' have a q-value of 1, they are exactly equally preferred. What to do? Firefox sends "Accept: text/html,application/xhtml+xml,application/xml;q=0.9, / ;q=0.8", it's true, which means we'll send text/html, not application/json, but this doesn't work for all browsers.
          Hide
          Robert Newson added a comment -

          The fix for jquery breaks authentication_redirect, find a solution that works everywhere for everyone. Also world peace and a pony.

          Show
          Robert Newson added a comment - The fix for jquery breaks authentication_redirect, find a solution that works everywhere for everyone. Also world peace and a pony.
          Hide
          Marcello Nuccio added a comment -

          "Accept: text/html, /"

          Both 'text/html' and 'application/json' have a q-value of 1,
          they are exactly equally preferred. What to do?

          Common sense says to me: get the best match first.

          "text/html" is an exact match, so I tend to say the client prefers it over the generic "/". How-to explain its inclusion otherwise, given that "/" includes "text/html"?

          Show
          Marcello Nuccio added a comment - "Accept: text/html, / " Both 'text/html' and 'application/json' have a q-value of 1, they are exactly equally preferred. What to do? Common sense says to me: get the best match first. "text/html" is an exact match, so I tend to say the client prefers it over the generic " / ". How-to explain its inclusion otherwise, given that " / " includes "text/html"?
          Hide
          Marcello Nuccio added a comment -

          RFC 2616 says the same thing I tried to say in my previous comment.

          From http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1

          Media ranges can be overridden by more specific media ranges or
          specific media types. If more than one media range applies to a given
          type, the most specific reference has precedence. For example,

          Accept: text/, text/html, text/html;level=1, */

          have the following precedence:

          1) text/html;level=1
          2) text/html
          3) text/*
          4) /

          IMHO this eliminates any ambiguity on "Accept: text/html, /".

          Show
          Marcello Nuccio added a comment - RFC 2616 says the same thing I tried to say in my previous comment. From http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.1 Media ranges can be overridden by more specific media ranges or specific media types. If more than one media range applies to a given type, the most specific reference has precedence. For example, Accept: text/ , text/html, text/html;level=1, */ have the following precedence: 1) text/html;level=1 2) text/html 3) text/* 4) / IMHO this eliminates any ambiguity on "Accept: text/html, / ".
          Hide
          Benoit Chesneau added a comment - - edited

          Rather than "reverting the change" we should introduce a complete check of accepted mime-types and also what are content-types accepted on each responses. Rather than specifically solving authentication redirection (which is imo a bad hack) we should take the 3 months we have before 1.2 to take the time for that. I will attach later today an editable list of expected response / API calls so we can eventually start on it.

          Anyway, changing Authorization header in settings, is enough most of the time. The application can already catch this authorization error and choose to use another page for authentication or just authenticate directly in ajax.
          And this setting was added was added to solve this specific case and allowing cookie authentication via ajax. We don't have really need for this redirection which was introduced as a convenience and I think it's perfectly safe on an api point of view to just remove it.

          A quick ex:
          https://github.com/couchapp/couchdb-login-jquery

          doesn't use this rediction.

          Show
          Benoit Chesneau added a comment - - edited Rather than "reverting the change" we should introduce a complete check of accepted mime-types and also what are content-types accepted on each responses. Rather than specifically solving authentication redirection (which is imo a bad hack) we should take the 3 months we have before 1.2 to take the time for that. I will attach later today an editable list of expected response / API calls so we can eventually start on it. Anyway, changing Authorization header in settings, is enough most of the time. The application can already catch this authorization error and choose to use another page for authentication or just authenticate directly in ajax. And this setting was added was added to solve this specific case and allowing cookie authentication via ajax. We don't have really need for this redirection which was introduced as a convenience and I think it's perfectly safe on an api point of view to just remove it. A quick ex: https://github.com/couchapp/couchdb-login-jquery doesn't use this rediction.
          Hide
          Marcello Nuccio added a comment -

          https://github.com/couchapp/couchdb-login-jquery does not work with password protected DBs, because the browser cannot download the couchapp.

          How can I get any Javascript run on the browser if I cannot get it first?

          authentication_redirect solves this specific use-case.

          Show
          Marcello Nuccio added a comment - https://github.com/couchapp/couchdb-login-jquery does not work with password protected DBs, because the browser cannot download the couchapp. How can I get any Javascript run on the browser if I cannot get it first? authentication_redirect solves this specific use-case.
          Hide
          Johannes J. Schmidt added a comment - - edited

          Hey come on, this is a very relevant bug. For me its a blocker to use 1.0.2 until this is fixed.

          My scenario:
          I have a CouchApp and give my users their own dbs, which they can secure if they want to.
          Behind a vhost I route example.com/myfairuser to a list function inside the db myfairuser (the list function generates the html page).
          Now imagine myfairuser is secured. How can you login there? The user opens example.com/myfairuser with the browser and sees

          {"error":"unauthorized","reason":"You are not authorized to access this db."}

          .
          Don't tell me the user has to return to example.com using the browser navigation after reading the nice message

          btw. It makes no difference in Chrome or Firefox.

          g jo

          Show
          Johannes J. Schmidt added a comment - - edited Hey come on, this is a very relevant bug. For me its a blocker to use 1.0.2 until this is fixed. My scenario: I have a CouchApp and give my users their own dbs, which they can secure if they want to. Behind a vhost I route example.com/myfairuser to a list function inside the db myfairuser (the list function generates the html page). Now imagine myfairuser is secured. How can you login there? The user opens example.com/myfairuser with the browser and sees {"error":"unauthorized","reason":"You are not authorized to access this db."} . Don't tell me the user has to return to example.com using the browser navigation after reading the nice message btw. It makes no difference in Chrome or Firefox. g jo
          Hide
          Marcello Nuccio added a comment -

          I think I have found a solution which can make everyone happy.

          If we do support "q" parameters in Accept headers, we can remove "authentication_redirect" option.

          Any browser I know of uses it properly, and gives to "text/html" higher preference. We only need to send the login html page as response to unauthorized clients if they give higher preference to "text/html". We do not need to redirect them to another page.

          "authentication_redirect" can become "authentication_page" to keep the possibility to change it. But that is all.

          As I have said, there is no ambiguity in "Accept: text/html, /", because the standard says clearly how to deal with it.

          What do you think?

          Show
          Marcello Nuccio added a comment - I think I have found a solution which can make everyone happy. If we do support "q" parameters in Accept headers, we can remove "authentication_redirect" option. Any browser I know of uses it properly, and gives to "text/html" higher preference. We only need to send the login html page as response to unauthorized clients if they give higher preference to "text/html". We do not need to redirect them to another page. "authentication_redirect" can become "authentication_page" to keep the possibility to change it. But that is all. As I have said, there is no ambiguity in "Accept: text/html, / ", because the standard says clearly how to deal with it. What do you think?
          Hide
          Robert Newson added a comment -

          I think IE8 still sends;

          Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/msword, /

          http://www.gethifi.com/blog/browser-rest-http-accept-headers

          Show
          Robert Newson added a comment - I think IE8 still sends; Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/msword, / http://www.gethifi.com/blog/browser-rest-http-accept-headers
          Hide
          Marcello Nuccio added a comment -

          ok, I give up

          However, the post you suggested gives also a simple solution:

          Frameworks can enhance performance by ignoring
          the Accept header and relying on '.xml'-like extensions.

          Is it applicable to CouchApps?

          Show
          Marcello Nuccio added a comment - ok, I give up However, the post you suggested gives also a simple solution: Frameworks can enhance performance by ignoring the Accept header and relying on '.xml'-like extensions. Is it applicable to CouchApps?
          Hide
          Filipe Manana added a comment -

          This should help in choosing the content-type the request prefers when it accepts multiple (explicit, via type/* or / and with or without q)

          https://github.com/mochi/mochiweb/pull/49

          Show
          Filipe Manana added a comment - This should help in choosing the content-type the request prefers when it accepts multiple (explicit, via type/* or / and with or without q) https://github.com/mochi/mochiweb/pull/49
          Hide
          Jason Smith added a comment -

          Great discussion. I also hope this fix goes into 1.2. Marcello, don't give up! You are speaking truth to power!

          Benoit, I agree the best thing is to identify the correct response to the variety of queries. If you see me on IRC, I volunteer to help you build the list. I suspect that once we clearly define the requirements, lo and behold, the solution will be to follow the standard.

          Filipe, your pull request will help, but it is slightly incorrect? The primary sort is the 'q' value but the secondary sort is the more specific type.

          foo/bar > foo/* > /

          For CouchDB, I would not think that will cause a bug because the only mistake it could make is for "Accept: /, text/html" which seems like a rare header value in the wild. OTOH, if IE8 really sends "bull/shit, more/shit, /" then CouchDB should probably treat that as text/html. (AJAX tools like jQuery modify the header, so Couch is pretty sure this is a direct query for a resource intended to be displayed.)

          Show
          Jason Smith added a comment - Great discussion. I also hope this fix goes into 1.2. Marcello, don't give up! You are speaking truth to power! Benoit, I agree the best thing is to identify the correct response to the variety of queries. If you see me on IRC, I volunteer to help you build the list. I suspect that once we clearly define the requirements, lo and behold, the solution will be to follow the standard. Filipe, your pull request will help, but it is slightly incorrect? The primary sort is the 'q' value but the secondary sort is the more specific type. foo/bar > foo/* > / For CouchDB, I would not think that will cause a bug because the only mistake it could make is for "Accept: / , text/html" which seems like a rare header value in the wild. OTOH, if IE8 really sends "bull/shit, more/shit, / " then CouchDB should probably treat that as text/html. (AJAX tools like jQuery modify the header, so Couch is pretty sure this is a direct query for a resource intended to be displayed.)
          Hide
          Filipe Manana added a comment -

          Hi Jason,

          The resulting list not only depends on the Q values but also on the order of the input given to it (when Qs are equal) and the specificity of the types. The caller must give full types as the parameter to the call. E.g. Req:accepted_content_types(["text/html", "application/json"])

          For that call, if the Accept header is ""text/;q=0.8, */;q=0.5" it will return ["text/html", "application/json"], meaning the client prefers text/html and then application/json. When the Accept header has both "type/subtype" and "type/*", it will give preference to type/subtype if it matches any type given in the input list, example:

          Req9 = mochiweb_request:new(nil, 'GET', "/foo",

          {1, 1}

          ,
          mochiweb_headers:make([

          {"Accept", "text/*;q=0.9, text/html;q=0.5, */*;q=0.7"}

          ])),
          ?assertEqual(["application/json", "text/html"],
          Req9:accepted_content_types(["text/html", "application/json"])).

          I guess this is what you were concerned with?

          Show
          Filipe Manana added a comment - Hi Jason, The resulting list not only depends on the Q values but also on the order of the input given to it (when Qs are equal) and the specificity of the types. The caller must give full types as the parameter to the call. E.g. Req:accepted_content_types( ["text/html", "application/json"] ) For that call, if the Accept header is ""text/ ;q=0.8, */ ;q=0.5" it will return ["text/html", "application/json"] , meaning the client prefers text/html and then application/json. When the Accept header has both "type/subtype" and "type/*", it will give preference to type/subtype if it matches any type given in the input list, example: Req9 = mochiweb_request:new(nil, 'GET', "/foo", {1, 1} , mochiweb_headers:make([ {"Accept", "text/*;q=0.9, text/html;q=0.5, */*;q=0.7"} ])), ?assertEqual( ["application/json", "text/html"] , Req9:accepted_content_types( ["text/html", "application/json"] )). I guess this is what you were concerned with?
          Hide
          Marcello Nuccio added a comment -

          Jason, I am still convinced that full support for Accept header is a good thing. But I am also convinced that good support for all clients is a good thing too.

          The patch of Filipe is very welcome, but maybe it is not enough. For example, if I do a GET on

          http://localhost:5984/db/_design/app/page.html

          I think it is reasonable to expect an HTML response without knowing what the Accept header is. It is reasonable because it works everywhere... well... I hope so...

          Show
          Marcello Nuccio added a comment - Jason, I am still convinced that full support for Accept header is a good thing. But I am also convinced that good support for all clients is a good thing too. The patch of Filipe is very welcome, but maybe it is not enough. For example, if I do a GET on http://localhost:5984/db/_design/app/page.html I think it is reasonable to expect an HTML response without knowing what the Accept header is. It is reasonable because it works everywhere... well... I hope so...
          Hide
          Robert Newson added a comment -

          It turns out that browsers typically send horribly broken Accept lines, so 'proper' content type negotiation will not work out in practice.

          I propose this, inspired by this post (https://github.com/rails/rails/commit/1310231c15742bf7d99e2f143d88b383c32782d3);

          1) if the Accept line is exactly 'application/json', we return json (and 401's instead of 302's).
          2) otherwise we assume the request is from a browser and return html (and 302's instead of 401's).

          The version of jquery we use does not append '/' so the above ought to work.

          Feedback is needed, please.

          Show
          Robert Newson added a comment - It turns out that browsers typically send horribly broken Accept lines, so 'proper' content type negotiation will not work out in practice. I propose this, inspired by this post ( https://github.com/rails/rails/commit/1310231c15742bf7d99e2f143d88b383c32782d3 ); 1) if the Accept line is exactly 'application/json', we return json (and 401's instead of 302's). 2) otherwise we assume the request is from a browser and return html (and 302's instead of 401's). The version of jquery we use does not append ' / ' so the above ought to work. Feedback is needed, please.
          Hide
          Noah Slater added a comment -

          I'm going to weigh in here, largely without context, to say that I am for a solution that, by and large, respects q values as they are intended to be used. Requiring clients to specify exactly one media type to get this working seems quite broken to me. It assumes that clients have the ability to set this so specifically. I am responding to Robert Newson's suggestion above, which I am -1 on for the moment. It is worth noting that when a client lists multiple media types with identical q values, and we able to satisfy a subset of them, it is perfectly reasonable to then choose which we respond with based on an internal set of preferences. That is, all things being equal in terms of client q values, we could prefer to send text/html over application/json or whatever.

          Show
          Noah Slater added a comment - I'm going to weigh in here, largely without context, to say that I am for a solution that, by and large, respects q values as they are intended to be used. Requiring clients to specify exactly one media type to get this working seems quite broken to me. It assumes that clients have the ability to set this so specifically. I am responding to Robert Newson's suggestion above, which I am -1 on for the moment. It is worth noting that when a client lists multiple media types with identical q values, and we able to satisfy a subset of them, it is perfectly reasonable to then choose which we respond with based on an internal set of preferences. That is, all things being equal in terms of client q values, we could prefer to send text/html over application/json or whatever.
          Hide
          Robert Newson added a comment -

          The mechanism you want to support, and indeed I wanted to support when I created this ticket, simply doesn't work in the real world. How can you ignore that?

          Show
          Robert Newson added a comment - The mechanism you want to support, and indeed I wanted to support when I created this ticket, simply doesn't work in the real world. How can you ignore that?
          Hide
          Noah Slater added a comment -

          Saying that q values do not work in the real world is false. They work in some situations, and don't work in other situations - like much of the web stack of technologies.The solution you've proposed might break compatibility for a completely different set of use cases. I think it might be a good idea to put some kind of table together, with each use case we care about against compatibility with each proposed solution. It would then be easy to visualise where we should be making the trade-offs.

          Show
          Noah Slater added a comment - Saying that q values do not work in the real world is false. They work in some situations, and don't work in other situations - like much of the web stack of technologies.The solution you've proposed might break compatibility for a completely different set of use cases. I think it might be a good idea to put some kind of table together, with each use case we care about against compatibility with each proposed solution. It would then be easy to visualise where we should be making the trade-offs.
          Hide
          Robert Newson added a comment -

          http://www.gethifi.com/blog/browser-rest-http-accept-headers

          IE sends: Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/msword, /

          Both text/html and application/json have the same preference value (1.0). Whichever way we jump, something goes wrong.

          The point of insisting on 'Accept: application/json' exactly is that only explicit calls from code, XHR, etc, can do it, it's unlike anything any browser sends.

          I don't much like the solution, but I don't see a better one right now.

          Show
          Robert Newson added a comment - http://www.gethifi.com/blog/browser-rest-http-accept-headers IE sends: Accept: image/jpeg, application/x-ms-application, image/gif, application/xaml+xml, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/msword, / Both text/html and application/json have the same preference value (1.0). Whichever way we jump, something goes wrong. The point of insisting on 'Accept: application/json' exactly is that only explicit calls from code, XHR, etc, can do it, it's unlike anything any browser sends. I don't much like the solution, but I don't see a better one right now.
          Hide
          Jason Smith added a comment -

          I humbly submit that the problem remains insufficiently clearly-defined to negotiate the answer.

          Top-of-my-head, some confounding factors, sorted randomly, by relevance.

          • Browsers have built-in Accept defaults
          • The jQuery that ships with Futon overrides these
          • Future jQuery versions, or alternative AJAX libraries, which developers require override these
          • Browsers and servers tend to become more standard over time, not less
          • Couch tends to show us the future, not the past (cf. HTTP, REST, Erlang, Javascript, replication; "I was into Erlang before it was cool" as the man says)
          • Wikipedia (body) and Google search (mind) have become self-aware (cf. the greatest film ever, "Ghost in the Shell"). This being has no corporeal form but is, in point of fact, omnipresent, omnipotent, and benevolent within cyberspace. Having a superior intellect, it was naturally drawn to CouchDB and made itself known to us by the name "Noah Slater." Cross swords with Noah Slater at your own peril.

          For an unrelated problem, I need answers to some of these questions myself, so I will try to post concrete data during the weekend, either on the wiki as a table, or in here. My hope is that the answer will be more clear then.

          Show
          Jason Smith added a comment - I humbly submit that the problem remains insufficiently clearly-defined to negotiate the answer. Top-of-my-head, some confounding factors, sorted randomly, by relevance. Browsers have built-in Accept defaults The jQuery that ships with Futon overrides these Future jQuery versions, or alternative AJAX libraries, which developers require override these Browsers and servers tend to become more standard over time, not less Couch tends to show us the future, not the past (cf. HTTP, REST, Erlang, Javascript, replication; "I was into Erlang before it was cool" as the man says) Wikipedia (body) and Google search (mind) have become self-aware (cf. the greatest film ever, "Ghost in the Shell"). This being has no corporeal form but is, in point of fact, omnipresent, omnipotent, and benevolent within cyberspace. Having a superior intellect, it was naturally drawn to CouchDB and made itself known to us by the name "Noah Slater." Cross swords with Noah Slater at your own peril. For an unrelated problem, I need answers to some of these questions myself, so I will try to post concrete data during the weekend, either on the wiki as a table, or in here. My hope is that the answer will be more clear then.
          Hide
          Marcello Nuccio added a comment -

          A question: is it needed sending a 302 when login is required?

          Maybe it is perfectly fine to always send a 401, if we respond with the correct content-type.

          All attachments in CouchDB have a content-type. Does it make sense to respond with a different content-type for this attachment?

          For example, if `index.html` has content-type set to `text/html`, is it reasonable to expect a response with this content-type for this attachment?

          I think yes. So, if I do a GET http://localhost:5984/db/_design/app/page.html, I expect to get a 401 with an HTML login page if authentication is needed, and a 200 with the contents of index.html if no authentication is needed.

          Show
          Marcello Nuccio added a comment - A question: is it needed sending a 302 when login is required? Maybe it is perfectly fine to always send a 401, if we respond with the correct content-type. All attachments in CouchDB have a content-type. Does it make sense to respond with a different content-type for this attachment? For example, if `index.html` has content-type set to `text/html`, is it reasonable to expect a response with this content-type for this attachment? I think yes. So, if I do a GET http://localhost:5984/db/_design/app/page.html , I expect to get a 401 with an HTML login page if authentication is needed, and a 200 with the contents of index.html if no authentication is needed.
          Hide
          Robert Newson added a comment -

          My understanding is that returning 401 causes all browsers to create a modal login dialog box that cannot be styled by the application. This is the entire reason we have this fiddly mechanism.

          Show
          Robert Newson added a comment - My understanding is that returning 401 causes all browsers to create a modal login dialog box that cannot be styled by the application. This is the entire reason we have this fiddly mechanism.
          Hide
          Noah Slater added a comment -

          "Both text/html and application/json have the same preference value (1.0). Whichever way we jump, something goes wrong."

          I disagree. If the q value is the same, we get to pick which one we want to serve up. In this instance, we assume it is a browser request, so we serve up HTML. What is the problem in only serving up JSON if application/json or application/* has a higher q value than the text/html or text/* media types? Is there a use case this does not work for?

          Also, Jason... 7 coolpoints for you, sir.

          Show
          Noah Slater added a comment - "Both text/html and application/json have the same preference value (1.0). Whichever way we jump, something goes wrong." I disagree. If the q value is the same, we get to pick which one we want to serve up. In this instance, we assume it is a browser request, so we serve up HTML. What is the problem in only serving up JSON if application/json or application/* has a higher q value than the text/html or text/* media types? Is there a use case this does not work for? Also, Jason... 7 coolpoints for you, sir.
          Hide
          Robert Newson added a comment -

          that flips the behavior back to 1.0.x style. It was flipped in 1.1.x to address an issue with jquery (iirc). Benoit made the change, he should chip in with a better explanation.

          Show
          Robert Newson added a comment - that flips the behavior back to 1.0.x style. It was flipped in 1.1.x to address an issue with jquery (iirc). Benoit made the change, he should chip in with a better explanation.
          Hide
          Marcello Nuccio added a comment - - edited

          "My understanding is that returning 401
          causes all browsers to create a modal login
          dialog box that cannot be styled by the application."

          Not really.
          You have to send a "WWW-Authenticate" header to trigger it.
          In CouchDB this is configurable with:

          [httpd]
          ; Uncomment next line to trigger basic-auth popup on unauthorized requests.
          ;WWW-Authenticate = Basic realm="administrator"

          By the way, I do know that WWW-Authenticate response-header is required in 401 response messages, but CouchDB is already breaking the rules, since by default it is disabled.

          Show
          Marcello Nuccio added a comment - - edited "My understanding is that returning 401 causes all browsers to create a modal login dialog box that cannot be styled by the application." Not really. You have to send a "WWW-Authenticate" header to trigger it. In CouchDB this is configurable with: [httpd] ; Uncomment next line to trigger basic-auth popup on unauthorized requests. ;WWW-Authenticate = Basic realm="administrator" By the way, I do know that WWW-Authenticate response-header is required in 401 response messages, but CouchDB is already breaking the rules, since by default it is disabled.
          Hide
          Randall Leeds added a comment -

          Can I propose another solution? Always send a 401 and not a 3xx. If an application (Futon being one such application) wishes to not show this dialog then it should issue requests with XHR and handle the 401 itself. It actually seems somewhat broken to me to try to redirect to Futon files and I'd prefer we work toward *de*coupling CouchDB from Futon rather than vice versa.

          Show
          Randall Leeds added a comment - Can I propose another solution? Always send a 401 and not a 3xx. If an application (Futon being one such application) wishes to not show this dialog then it should issue requests with XHR and handle the 401 itself. It actually seems somewhat broken to me to try to redirect to Futon files and I'd prefer we work toward *de*coupling CouchDB from Futon rather than vice versa.
          Hide
          Marcello Nuccio added a comment -

          @Randall, this is almost exactly what I am trying to say.
          The only problem is: what to put in the 401 response?

          Right now, in CouchDB-1.1, if I do GET for an attachment with content-type 'text/html', the response could have content-type 'application/json'. This breaks the Couchapps hosted on password protected DB, because there's no easy way for the browser to get an HTML login page, when authentication is required.

          My solution is: if the client is requesting a 'text/html', give him a 'text/html', even if the Accept header says that 'application/json' has higher priority.

          This does make sense, because it is exactly what I do if authentication is not needed.

          Show
          Marcello Nuccio added a comment - @Randall, this is almost exactly what I am trying to say. The only problem is: what to put in the 401 response? Right now, in CouchDB-1.1, if I do GET for an attachment with content-type 'text/html', the response could have content-type 'application/json'. This breaks the Couchapps hosted on password protected DB, because there's no easy way for the browser to get an HTML login page, when authentication is required. My solution is: if the client is requesting a 'text/html', give him a 'text/html', even if the Accept header says that 'application/json' has higher priority. This does make sense, because it is exactly what I do if authentication is not needed.
          Hide
          James Howe added a comment -

          Just FYI, the JSONView Firefox extension gives you (optionally) an Accept of:
          text/html,application/xhtml+xml,application/xml;q=0.9,/;q=0.8,application/json

          I would guess many people would have it, as it lets you use Couch APIs directly without struggling to match braces and quotes.

          Show
          James Howe added a comment - Just FYI, the JSONView Firefox extension gives you (optionally) an Accept of: text/html,application/xhtml+xml,application/xml;q=0.9, / ;q=0.8,application/json I would guess many people would have it, as it lets you use Couch APIs directly without struggling to match braces and quotes.
          Hide
          Robert Newson added a comment -

          Bump.

          I'd like to resolve this issue as 'won't fix' and move forward with the 1.1.1 release. Any objections?

          Show
          Robert Newson added a comment - Bump. I'd like to resolve this issue as 'won't fix' and move forward with the 1.1.1 release. Any objections?
          Hide
          matthew o'gorman added a comment -

          It's unfortunate if this is marked as won't fixed as it will make secure couchapps far more difficult to setup. The current options as I understand it require you having a http proxy in place as well as a second db that is set to not allow writes that can connect to the _users table to allow the user to log in and then redirect back to the main real app. I realize that there seems to be no clean solution to the problem but it does seem important at least for the people who want to write apps that are direct couchdb applications.

          Show
          matthew o'gorman added a comment - It's unfortunate if this is marked as won't fixed as it will make secure couchapps far more difficult to setup. The current options as I understand it require you having a http proxy in place as well as a second db that is set to not allow writes that can connect to the _users table to allow the user to log in and then redirect back to the main real app. I realize that there seems to be no clean solution to the problem but it does seem important at least for the people who want to write apps that are direct couchdb applications.
          Hide
          Robert Newson added a comment -

          Not happening in 1.1.1

          Show
          Robert Newson added a comment - Not happening in 1.1.1
          Hide
          Noah Slater added a comment -

          Fair enough, but it should remain open.

          Show
          Noah Slater added a comment - Fair enough, but it should remain open.
          Hide
          Randall Leeds added a comment -

          These look related.

          Show
          Randall Leeds added a comment - These look related.
          Hide
          Ari Najarian added a comment -

          Hi folks,

          I was shocked, and quite happy, to discover a thread that discusses the obscure issue I'm having. This alone prompted me to sign up to the forum so I could upvote this issue and watch it.

          Like Marcello and Johannes, I'm trying to work on a secure couchapp, and am coming up against the same problem. If I restrict access to particular database to authenticated readers only, then when anyone navigates to the design document, they get a JSON response instead of a redirect.

          Jason mentioned that the problem was insufficiently defined to move forward. As I see it, the problem is quite simple : right now, one can either create a couchapp that sits upon a database that anonymous users can access, OR they can create a secure document repository that only non-browser clients can interact with. However, there is no way to create a couchapp that interacts with a secure database, as there's no way to authenticate the user if they hit up the application.

          I don't want anonymous users to be able to access the information in my database through REST. I don't know a damned thing about HTTP headers, responses or content-types. I'm hoping this is an easy fix that will be pushed out to the internet soon. From the pros in this forum, any idea how long I may have to wait to see this bug resolved?

          Show
          Ari Najarian added a comment - Hi folks, I was shocked, and quite happy, to discover a thread that discusses the obscure issue I'm having. This alone prompted me to sign up to the forum so I could upvote this issue and watch it. Like Marcello and Johannes, I'm trying to work on a secure couchapp, and am coming up against the same problem. If I restrict access to particular database to authenticated readers only, then when anyone navigates to the design document, they get a JSON response instead of a redirect. Jason mentioned that the problem was insufficiently defined to move forward. As I see it, the problem is quite simple : right now, one can either create a couchapp that sits upon a database that anonymous users can access, OR they can create a secure document repository that only non-browser clients can interact with. However, there is no way to create a couchapp that interacts with a secure database, as there's no way to authenticate the user if they hit up the application. I don't want anonymous users to be able to access the information in my database through REST. I don't know a damned thing about HTTP headers, responses or content-types. I'm hoping this is an easy fix that will be pushed out to the internet soon. From the pros in this forum, any idea how long I may have to wait to see this bug resolved?
          Hide
          Marcello Nuccio added a comment -

          Ari, there's no easy fix. Luckily, you have an easy workaround: put the data in a secure db, and the application in a read-only db (or HTTP server).

          Show
          Marcello Nuccio added a comment - Ari, there's no easy fix. Luckily, you have an easy workaround: put the data in a secure db, and the application in a read-only db (or HTTP server).
          Hide
          Johannes J. Schmidt added a comment -

          As Marcello mentioned, the only possible solution by now is to depend on an architecture where the application loads the data via AJAX.
          What I would love to do is to generate plain html (and other formats) out of the data via list functions, so that the data is accessible through clients without JavaScript (eg. search engines).
          Currently I build on the former but I hope I can provide static html versions of the content later when this issue is resolved.

          Show
          Johannes J. Schmidt added a comment - As Marcello mentioned, the only possible solution by now is to depend on an architecture where the application loads the data via AJAX. What I would love to do is to generate plain html (and other formats) out of the data via list functions, so that the data is accessible through clients without JavaScript (eg. search engines). Currently I build on the former but I hope I can provide static html versions of the content later when this issue is resolved.
          Hide
          Johannes J. Schmidt added a comment -

          Also consider the performance boost sending server side (pre-) rendered html on first page access and then using AJAX for future requests. This results in a great user experience (I think google uses this technology widely). It is possible (when this issue would be resolved) with list functions using only CouchDB and no other server software.
          Not depending on any other components than CouchDB is one of the things I love CouchDB for.

          Show
          Johannes J. Schmidt added a comment - Also consider the performance boost sending server side (pre-) rendered html on first page access and then using AJAX for future requests. This results in a great user experience (I think google uses this technology widely). It is possible (when this issue would be resolved) with list functions using only CouchDB and no other server software. Not depending on any other components than CouchDB is one of the things I love CouchDB for.
          Hide
          Ari Najarian added a comment -

          Marcello, Johannes, thanks for your suggestions. I'm actively looking for a workaround for the time being.

          I had considered the dual-database solution, where secure data is locked down in one db, and the couchapp design document sits on a separate database. I had a few (4) misgivings about this approach, though.

          1. If I'm using AJAX to pull in data from the secure database, I would either have to use jQuery.couch.js to authenticate (meaning the credentials are stored in the code), or I'd have to provide a login prompt to authenticate the user and retrieve a session-based cookie so that requests to the secure database return documents.

          The former case is not secure, as anybody can view source and pull the login credentials that I'm using to unlock the database. The latter might work, but I struggle with how to handle the situation when the cookie expires. I'm building a mobile couchapp, so the user might return after several hours to check for new data. When the AJAX request is fired off, we come up against the same problem of a JSON response being returned. Are you suggesting that I intercept this response and display a prompt to the user to re-authenticate?

          2. Perhaps most importantly, if the entire couchapp is built using AJAX, am I able to benefit from any of the view, shows, templates and partials that are features of couchDB? Design documents, as far as I understand, are walled gardens, so relative paths end up at the root of _design/couchapp. Is there a way, without using absolute paths, to reference a show or list function in another design document in another database? Absolute paths to a show function would be problematic: if the couchapp is ever replicated to server B, it would still look for the show function on server A.

          3. Related to the previous point, using AJAX to re-authenticate and pull in data from a separate design document would basically mean that JS is now acting as another layer on the stack - not the most elegant solution, as the whole architecture of couchapp is supposed to be self-contained.

          4. Writing a jQuery mobile app that lives in a database separate from the actual documents it serves would require me to replicate two separate databases to a new server, correct? If so, this is again inelegant, as the couchapp isn't self-contained anymore.

          I'm not bringing up these points to whine or complain. I'm just trying to make sure I move forward in the best way possible. My understanding of cocuhDB is still a little shaky, so if there are any misconceptions in my statements above, I'd appreciate clarification. As it stands, I'm going to migrate my jQuery mobile couchapp to a separate design document in a separate database, and make absolute calls to the views and shows I build in a sibling database's design document in order to populate the pages of the couchapp.

          I hope that makes sense. Please stop me if I'm doing it wrong!

          Show
          Ari Najarian added a comment - Marcello, Johannes, thanks for your suggestions. I'm actively looking for a workaround for the time being. I had considered the dual-database solution, where secure data is locked down in one db, and the couchapp design document sits on a separate database. I had a few (4) misgivings about this approach, though. 1. If I'm using AJAX to pull in data from the secure database, I would either have to use jQuery.couch.js to authenticate (meaning the credentials are stored in the code), or I'd have to provide a login prompt to authenticate the user and retrieve a session-based cookie so that requests to the secure database return documents. The former case is not secure, as anybody can view source and pull the login credentials that I'm using to unlock the database. The latter might work, but I struggle with how to handle the situation when the cookie expires. I'm building a mobile couchapp, so the user might return after several hours to check for new data. When the AJAX request is fired off, we come up against the same problem of a JSON response being returned. Are you suggesting that I intercept this response and display a prompt to the user to re-authenticate? 2. Perhaps most importantly, if the entire couchapp is built using AJAX, am I able to benefit from any of the view, shows, templates and partials that are features of couchDB? Design documents, as far as I understand, are walled gardens, so relative paths end up at the root of _design/couchapp. Is there a way, without using absolute paths, to reference a show or list function in another design document in another database? Absolute paths to a show function would be problematic: if the couchapp is ever replicated to server B, it would still look for the show function on server A. 3. Related to the previous point, using AJAX to re-authenticate and pull in data from a separate design document would basically mean that JS is now acting as another layer on the stack - not the most elegant solution, as the whole architecture of couchapp is supposed to be self-contained. 4. Writing a jQuery mobile app that lives in a database separate from the actual documents it serves would require me to replicate two separate databases to a new server, correct? If so, this is again inelegant, as the couchapp isn't self-contained anymore. I'm not bringing up these points to whine or complain. I'm just trying to make sure I move forward in the best way possible. My understanding of cocuhDB is still a little shaky, so if there are any misconceptions in my statements above, I'd appreciate clarification. As it stands, I'm going to migrate my jQuery mobile couchapp to a separate design document in a separate database, and make absolute calls to the views and shows I build in a sibling database's design document in order to populate the pages of the couchapp. I hope that makes sense. Please stop me if I'm doing it wrong!
          Hide
          Jan Lehnardt added a comment -

          bump to 1.3.x

          Show
          Jan Lehnardt added a comment - bump to 1.3.x
          Hide
          Robert Newson added a comment -

          Vote to bump to an unspecified future release given the lack of consensus on a solution here.

          Show
          Robert Newson added a comment - Vote to bump to an unspecified future release given the lack of consensus on a solution here.
          Hide
          Jan Lehnardt added a comment -

          +1

          Show
          Jan Lehnardt added a comment - +1
          Hide
          Dave Cottlehuber added a comment -

          As there is no clear path forwards despite much discussion; I'm closing this until somebody comes up with a Fix For All Browsers And RESTful Services.

          Show
          Dave Cottlehuber added a comment - As there is no clear path forwards despite much discussion; I'm closing this until somebody comes up with a Fix For All Browsers And RESTful Services.

            People

            • Assignee:
              Unassigned
              Reporter:
              Robert Newson
            • Votes:
              9 Vote for this issue
              Watchers:
              11 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development