CouchDB
  1. CouchDB
  2. COUCHDB-1100

Server doesn't release HTTP connection when client queries a currently updating view and times out

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: 1.0.2
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      If a client queries a view that is currently updating (without stale=ok), and then closes the connection after a timeout, the resources on the server side are not released (until the view update completes).

      If the view update takes a long time, the server will eventually run out of file descriptors (or if the file descriptor limit is very large, runs out of something else; not quite sure what).

      Steps to reproduce the problem:

      1) set "ulimit -n" for couchdb to 1024 (often the default). Using a much larger limit seems to result in different behavior (possibly CouchDB runs out of some other resource before file descriptors; not sure what).

      2) create a database with a view and sufficient number of docs so that generating the view from scratch takes a lot of time

      3) query the view repeatedly with a short timeout: for example in bash:
      while true; do curl -m1 'http://127.0.0.1:5984/grande/_design/testviews/_view/byDate1?limit=1'; done

      4) monitor the number of sockets used with e.g. "netstat -an | grep 5984 | grep CLOSE_WAIT". This should increase all the time.

      5) roughly when the number hits 1K, couchdb stops responding completely, and there's large number of error messages written to the log file.

        Activity

        Hide
        Paul Joseph Davis added a comment -

        Sounds like mochiweb/gen_tcp/something isn't raising any sort of message when the socket closes. Though I can't say that I could really expect it unless it was doing some sort of active monitoring on the file descriptor, which seems unlikely given that I'm pretty sure mochiweb uses the whole active_once flag (IIRC).

        Not entirely certain what the best solution to this would be other than maybe a maximum request time? But that sounds an awful lot like just hitting it with a hammer.

        Show
        Paul Joseph Davis added a comment - Sounds like mochiweb/gen_tcp/something isn't raising any sort of message when the socket closes. Though I can't say that I could really expect it unless it was doing some sort of active monitoring on the file descriptor, which seems unlikely given that I'm pretty sure mochiweb uses the whole active_once flag (IIRC). Not entirely certain what the best solution to this would be other than maybe a maximum request time? But that sounds an awful lot like just hitting it with a hammer.
        Hide
        Pasi Eronen added a comment -

        The client usually has some maximum time it's willing to wait for the response (=finite socket timeout in HTTP code, if nothing else). Perhaps the client could tell the server this value (with e.g. a request parameter or a custom HTTP header)?

        Show
        Pasi Eronen added a comment - The client usually has some maximum time it's willing to wait for the response (=finite socket timeout in HTTP code, if nothing else). Perhaps the client could tell the server this value (with e.g. a request parameter or a custom HTTP header)?
        Hide
        Randall Leeds added a comment -

        Could you try adding

        {keepalive, true}

        to the list in socket_options, httpd section of the configuration?

        Show
        Randall Leeds added a comment - Could you try adding {keepalive, true} to the list in socket_options, httpd section of the configuration?

          People

          • Assignee:
            Unassigned
            Reporter:
            Pasi Eronen
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development