Details

    • Type: New Feature
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Replication
    • Labels:
      None

      Activity

      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit b7b2ac3663fc653ece65ab601d48333d07aac878 in couchdb-couch-replicator's branch refs/heads/3010-add-rate-limiter from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=b7b2ac3 ]

      Add rate limiter for replication

      For a remote replication request, we add in support for rate limiting.
      We add in 4 new replication options:

      src_rate_limit
      src_rate_period
      target_rate_limit
      target_rate_period

      These define the maximum number of requests per interval,
      i.e limit = 10, period = 5 would be a rate of 2 requests per second.

      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit b7b2ac3663fc653ece65ab601d48333d07aac878 in couchdb-couch-replicator's branch refs/heads/3010-add-rate-limiter from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=b7b2ac3 ] Add rate limiter for replication For a remote replication request, we add in support for rate limiting. We add in 4 new replication options: src_rate_limit src_rate_period target_rate_limit target_rate_period These define the maximum number of requests per interval, i.e limit = 10, period = 5 would be a rate of 2 requests per second. COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit c6e891d26879bdaef9408196f436769e69e5e58f in couchdb-couch-replicator's branch refs/heads/3010-handle-429 from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=c6e891d ]

      Add exponential backoff for 429 errors.

      When we encounter a 429, we retry with a different set of retries and
      timeout. This will theoretically reduce client replication overload.
      When 429s have stopped, it's possible that a 500 error could occur.
      Then the retry mechanism should go back to the original way for
      backwards compatibility.

      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit c6e891d26879bdaef9408196f436769e69e5e58f in couchdb-couch-replicator's branch refs/heads/3010-handle-429 from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=c6e891d ] Add exponential backoff for 429 errors. When we encounter a 429, we retry with a different set of retries and timeout. This will theoretically reduce client replication overload. When 429s have stopped, it's possible that a 500 error could occur. Then the retry mechanism should go back to the original way for backwards compatibility. COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit 665b0a7f94404c1af5d338ad2a8345bc94174db8 in couchdb-couch-replicator's branch refs/heads/3010-handle-429 from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=665b0a7 ]

      Clean up syntax

      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit 665b0a7f94404c1af5d338ad2a8345bc94174db8 in couchdb-couch-replicator's branch refs/heads/3010-handle-429 from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=665b0a7 ] Clean up syntax COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit 7f0f3d7e883be81946f1dbc0108649fc81772964 in couchdb-couch-replicator's branch refs/heads/3010-handle-429 from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=7f0f3d7 ]

      Add random for next wait in maybe_retry

      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit 7f0f3d7e883be81946f1dbc0108649fc81772964 in couchdb-couch-replicator's branch refs/heads/3010-handle-429 from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=7f0f3d7 ] Add random for next wait in maybe_retry COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit 81d55441f7b9da0c639fb2f02492f1af6d733eb8 in couchdb-couch-replicator's branch refs/heads/3010-handle-429 from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=81d5544 ]

      Change back_off to backoff

      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit 81d55441f7b9da0c639fb2f02492f1af6d733eb8 in couchdb-couch-replicator's branch refs/heads/3010-handle-429 from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=81d5544 ] Change back_off to backoff COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit 80e9578228365277ee20d0b02a1f4bd6514c5f12 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=80e9578 ]

      Handle 429

      When we encounter a 429, we retry with a different set of retries and
      timeout. This will theoretically reduce client replication overload.
      When 429s have stopped, it's possible that a 500 error could occur.
      Then the retry mechanism should go back to the original way for
      backwards compatibility.

      BugzId:60007
      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit 80e9578228365277ee20d0b02a1f4bd6514c5f12 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=80e9578 ] Handle 429 When we encounter a 429, we retry with a different set of retries and timeout. This will theoretically reduce client replication overload. When 429s have stopped, it's possible that a 500 error could occur. Then the retry mechanism should go back to the original way for backwards compatibility. BugzId:60007 COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit 2db0d7f2a697c7e4aae9ca0ccb1658591a7579c5 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=2db0d7f ]

      Retry when connection_closed is received during a streamed response

      The changes_reader uses a streamed response. During the stream, it's
      possible to receive a connection_closed error due to timeouts or
      network issues. We simply retry the request because for streamed
      responses a connection must be established first in order for the
      stream to begin. So if the network is truly down, the initial request
      will fail and the code path will go through the normal retry clause
      which decrements the number of retries. This way we won't be stuck
      forever if it's an actual network issue.

      BugzId: 70400
      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit 2db0d7f2a697c7e4aae9ca0ccb1658591a7579c5 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=2db0d7f ] Retry when connection_closed is received during a streamed response The changes_reader uses a streamed response. During the stream, it's possible to receive a connection_closed error due to timeouts or network issues. We simply retry the request because for streamed responses a connection must be established first in order for the stream to begin. So if the network is truly down, the initial request will fail and the code path will go through the normal retry clause which decrements the number of retries. This way we won't be stuck forever if it's an actual network issue. BugzId: 70400 COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit a51561c1e07e63371cef4f797f063028b4c61999 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=a51561c ]

      Fix timeout clause in backoff retry

      The second clause for a timeout will never be reached because the first
      will always match before the second clause. Switching the clauses to
      fix this.

      BugzId:70400
      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit a51561c1e07e63371cef4f797f063028b4c61999 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=a51561c ] Fix timeout clause in backoff retry The second clause for a timeout will never be reached because the first will always match before the second clause. Switching the clauses to fix this. BugzId:70400 COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit ada53cbff71c06c924ad4044241d3e01c888e133 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=ada53cb ]

      Make backoff macros configurable

      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit ada53cbff71c06c924ad4044241d3e01c888e133 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=ada53cb ] Make backoff macros configurable COUCHDB-3010
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit cb41bacb2a06613649df46d62249afebda42b8c0 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=cb41bac ]

      Merge branch '3010-port-429' into apache

      COUCHDB-3010

      Show
      jira-bot ASF subversion and git services added a comment - Commit cb41bacb2a06613649df46d62249afebda42b8c0 in couchdb-couch-replicator's branch refs/heads/master from Tony Sun [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=cb41bac ] Merge branch '3010-port-429' into apache COUCHDB-3010

        People

        • Assignee:
          Unassigned
          Reporter:
          tonysun83 Tony Sun
        • Votes:
          0 Vote for this issue
          Watchers:
          2 Start watching this issue

          Dates

          • Created:
            Updated:

            Development