CouchDB
  1. CouchDB
  2. COUCHDB-1901

Replicator needs to retry multipart requests atomically

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.5.0
    • Component/s: Replication
    • Labels:
      None

      Description

      couch_replicator_httpc's retry mechanism interacts poorly with the multipart requests we use to transfer attachments during replication. In the case of PUTs, the replicator will retry a request even though it has already streamed part of the request body. For GETs, it can potentially interleave body data from the original response and the retried one.

      I put together a fix at https://github.com/cloudant/couch_replicator/pull/36 that disables the default httpc client retry mechanism for multipart requests and manages the retry logic "atomically"; i.e., it retries the original GET request if the GET or PUT fails, and it ensures that the original GET request is properly terminated during the retry. I'll push a version of that patch to the Apache repo now.

        Activity

        Adam Kocoloski created issue -
        Adam Kocoloski made changes -
        Field Original Value New Value
        Assignee Adam Kocoloski [ kocolosk ]
        Fix Version/s 1.5.0 [ 12324309 ]
        Hide
        Adam Kocoloski added a comment -

        I pushed a pull request at https://github.com/apache/couchdb/pull/95 which includes the work from the aforementioned Cloudant PR as well as some earlier fixes by Paul Joseph Davis regarding connection handling in the multipart workflow.

        Show
        Adam Kocoloski added a comment - I pushed a pull request at https://github.com/apache/couchdb/pull/95 which includes the work from the aforementioned Cloudant PR as well as some earlier fixes by Paul Joseph Davis regarding connection handling in the multipart workflow.
        Hide
        ASF subversion and git services added a comment -

        Commit df0423b5d5c35170626cfc0c10216d2e72e3a83e in branch refs/heads/1901-atomic-multipart-retries from Adam Kocoloski
        [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=df0423b ]

        Bubble up missing_stub exceptions for proper retry

        A missing_stub exception is special in that the replicator retries the
        request without the atts_since optimmization.

        COUCHDB-1901

        Show
        ASF subversion and git services added a comment - Commit df0423b5d5c35170626cfc0c10216d2e72e3a83e in branch refs/heads/1901-atomic-multipart-retries from Adam Kocoloski [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=df0423b ] Bubble up missing_stub exceptions for proper retry A missing_stub exception is special in that the replicator retries the request without the atts_since optimmization. COUCHDB-1901
        Hide
        ASF subversion and git services added a comment -

        Commit df0423b5d5c35170626cfc0c10216d2e72e3a83e in branch refs/heads/master from Adam Kocoloski
        [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=df0423b ]

        Bubble up missing_stub exceptions for proper retry

        A missing_stub exception is special in that the replicator retries the
        request without the atts_since optimmization.

        COUCHDB-1901

        Show
        ASF subversion and git services added a comment - Commit df0423b5d5c35170626cfc0c10216d2e72e3a83e in branch refs/heads/master from Adam Kocoloski [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=df0423b ] Bubble up missing_stub exceptions for proper retry A missing_stub exception is special in that the replicator retries the request without the atts_since optimmization. COUCHDB-1901
        Hide
        ASF subversion and git services added a comment -

        Commit eb137e3f8c3b618020b6d1032bcb35238d93c2e1 in branch refs/heads/master from Dirkjan Ochtman
        [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=eb137e3 ]

        Add note about COUCHDB-1901 to changelog.

        Show
        ASF subversion and git services added a comment - Commit eb137e3f8c3b618020b6d1032bcb35238d93c2e1 in branch refs/heads/master from Dirkjan Ochtman [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=eb137e3 ] Add note about COUCHDB-1901 to changelog.
        Jan Lehnardt made changes -
        Status Open [ 1 ] Resolved [ 5 ]
        Resolution Fixed [ 1 ]
        Jan Lehnardt made changes -
        Status Resolved [ 5 ] Closed [ 6 ]
        Hide
        ASF subversion and git services added a comment -

        Commit 15d75c360a5c27cc0a0d9f03a707ca5b08e31c88 in couchdb-couch-replicator's branch refs/heads/1843-feature-bigcouch from Adam Kocoloski
        [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=15d75c3 ]

        Bubble up missing_stub exceptions for proper retry

        A missing_stub exception is special in that the replicator retries the
        request without the atts_since optimmization.

        COUCHDB-1901

        Show
        ASF subversion and git services added a comment - Commit 15d75c360a5c27cc0a0d9f03a707ca5b08e31c88 in couchdb-couch-replicator's branch refs/heads/1843-feature-bigcouch from Adam Kocoloski [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=15d75c3 ] Bubble up missing_stub exceptions for proper retry A missing_stub exception is special in that the replicator retries the request without the atts_since optimmization. COUCHDB-1901
        Hide
        ASF subversion and git services added a comment -

        Commit 21ab541c37bcbdb0a4ac836ab5cd7243b9ced3e2 in couchdb-couch-replicator's branch refs/heads/1843-feature-bigcouch from Adam Kocoloski
        [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=21ab541 ]

        Bubble up missing_stub exceptions for proper retry

        A missing_stub exception is special in that the replicator retries the
        request without the atts_since optimmization.

        COUCHDB-1901

        Show
        ASF subversion and git services added a comment - Commit 21ab541c37bcbdb0a4ac836ab5cd7243b9ced3e2 in couchdb-couch-replicator's branch refs/heads/1843-feature-bigcouch from Adam Kocoloski [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=21ab541 ] Bubble up missing_stub exceptions for proper retry A missing_stub exception is special in that the replicator retries the request without the atts_since optimmization. COUCHDB-1901
        Transition Time In Source Status Execution Times Last Executer Last Execution Date
        Open Open Resolved Resolved
        1d 15h 47m 1 Jan Lehnardt 04/Oct/13 08:33
        Resolved Resolved Closed Closed
        8s 1 Jan Lehnardt 04/Oct/13 08:33

          People

          • Assignee:
            Adam Kocoloski
            Reporter:
            Adam Kocoloski
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development