Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-2833

Replicator client doesn't handle un-expectedly closed pipelined connections well

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 2.0.0
    • Component/s: Replication
    • Labels:
      None

      Description

      This was found investigating the failure of replication tests. Specifically couch_replicator_large_atts_tests, the

      {local, remote}

      sub-case.

      The test sets up push replications from local to remote.

      Replication workers have more than 1 document larger than MAX_BULK_ATT_SIZE=64K. They start pushing them to the target, using a keep-alive connection (default for HTTP 1.1), the first few pipelined requests will go through using the same connection, then server will accept the first PUT to …/docid?edits=false, then return Connection:close and close the connection after the 201 Created result. Workers don't expect that, and try to do another PUT on same connection. And then crash on ibrowser's connection_closing error, which they don't handle. That causes the whole async replication process to exit.

      Potentially there are 2 issues.

      couch_replicator_http layer needs to handle this case better. On closing error, shut down the socket quickly and then retry. (Not shutting it down and retrying means retrying for at least 5 or so seconds until something cleans up that connection state).

      Adding this clause to couch_replicator_httpc.erl seems to do the trick:

      process_response({error,connection_closing}, Worker, HttpDb, Params, _Cb)->
          couch_log:notice("Connection closed by server. Closing the socket and trying again",[]),
          ibrowse_http_client:stop(Worker),
          throw({retry, HttpDb, Params});
      

      Another issue is to make the server not close the connection after first PUT to .../db/docid?new_edits=false when using pipeline connections. See COUCHDB-2834 for more information on that issue.

        Attachments

          Activity

            People

            • Assignee:
              kzx kzx
              Reporter:
              vatamane Nick Vatamaniuc
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: