CouchDB
  1. CouchDB
  2. COUCHDB-1901

Replicator needs to retry multipart requests atomically

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: Replication
    • Labels:
      None

      Description

      couch_replicator_httpc's retry mechanism interacts poorly with the multipart requests we use to transfer attachments during replication. In the case of PUTs, the replicator will retry a request even though it has already streamed part of the request body. For GETs, it can potentially interleave body data from the original response and the retried one.

      I put together a fix at https://github.com/cloudant/couch_replicator/pull/36 that disables the default httpc client retry mechanism for multipart requests and manages the retry logic "atomically"; i.e., it retries the original GET request if the GET or PUT fails, and it ensures that the original GET request is properly terminated during the retry. I'll push a version of that patch to the Apache repo now.

        Activity

        Jan Lehnardt made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Jan Lehnardt made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Hide
        ASF subversion and git services added a comment -

        Commit eb137e3f8c3b618020b6d1032bcb35238d93c2e1 in branch refs/heads/master from Dirkjan Ochtman
        [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=eb137e3 ]

        Add note about COUCHDB-1901 to changelog.

        Show
        ASF subversion and git services added a comment - Commit eb137e3f8c3b618020b6d1032bcb35238d93c2e1 in branch refs/heads/master from Dirkjan Ochtman [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=eb137e3 ] Add note about COUCHDB-1901 to changelog.
        Hide
        ASF subversion and git services added a comment -

        Commit df0423b5d5c35170626cfc0c10216d2e72e3a83e in branch refs/heads/master from Adam Kocoloski
        [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=df0423b ]

        Bubble up missing_stub exceptions for proper retry

        A missing_stub exception is special in that the replicator retries the
        request without the atts_since optimmization.

        COUCHDB-1901

        Show
        ASF subversion and git services added a comment - Commit df0423b5d5c35170626cfc0c10216d2e72e3a83e in branch refs/heads/master from Adam Kocoloski [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=df0423b ] Bubble up missing_stub exceptions for proper retry A missing_stub exception is special in that the replicator retries the request without the atts_since optimmization. COUCHDB-1901
        Hide
        ASF subversion and git services added a comment -

        Commit df0423b5d5c35170626cfc0c10216d2e72e3a83e in branch refs/heads/1901-atomic-multipart-retries from Adam Kocoloski
        [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=df0423b ]

        Bubble up missing_stub exceptions for proper retry

        A missing_stub exception is special in that the replicator retries the
        request without the atts_since optimmization.

        COUCHDB-1901

        Show
        ASF subversion and git services added a comment - Commit df0423b5d5c35170626cfc0c10216d2e72e3a83e in branch refs/heads/1901-atomic-multipart-retries from Adam Kocoloski [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=df0423b ] Bubble up missing_stub exceptions for proper retry A missing_stub exception is special in that the replicator retries the request without the atts_since optimmization. COUCHDB-1901
        Hide
        Adam Kocoloski added a comment -

        I pushed a pull request at https://github.com/apache/couchdb/pull/95 which includes the work from the aforementioned Cloudant PR as well as some earlier fixes by Paul Joseph Davis regarding connection handling in the multipart workflow.

        Show
        Adam Kocoloski added a comment - I pushed a pull request at https://github.com/apache/couchdb/pull/95 which includes the work from the aforementioned Cloudant PR as well as some earlier fixes by Paul Joseph Davis regarding connection handling in the multipart workflow.
        Adam Kocoloski made changes -
        Field Original Value New Value
        Assignee Adam Kocoloski [ kocolosk ]
        Fix Version/s 1.5.0 [ 12324309 ]
        Adam Kocoloski created issue -

          People

          • Assignee:
            Adam Kocoloski
            Reporter:
            Adam Kocoloski
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development