Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-2964

Investigate switching replicator manager change feeds to using "normal" instead of "longpoll"

    Details

    • Type: Improvement
    • Status: Reopened
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 2.1
    • Component/s: Replication
    • Labels:
      None

      Activity

      Hide
      vatamane Nick Vatamaniuc added a comment - - edited

      Currently 2.0 replicator manager uses "longpoll" change feeds to get replicator database changes. Since replicator manager uses only change feeds from local shards, and not going though the fabric layer, it is interesting to investigate switching to "normal" mode.

      Details:

      Previous couch_replicator's change feeds go through the fabric layer, so longpoll is needed because the set of shards which signaled the update, might not be the shards change feed will hit first when they are spawned. In that case if "normal" mode is used, then the change feed could immediately return without waiting for the new change to propagate via internal replication. Longpoll mode ensures there is a wait for shards to receive the change before stopping.

      CouchDB 2.0 replicator manager uses change feeds on individual shards. It seems even though "longpoll" mode will still work correctly (replicator manager is notified there is an update to the database, then gets changes from last checkpoint, and then saves the new checkpoint), "normal" mode might be more semantically correct, as effectively "longpoll" is used as normal – it never waits on changes.

      Show
      vatamane Nick Vatamaniuc added a comment - - edited Currently 2.0 replicator manager uses "longpoll" change feeds to get replicator database changes. Since replicator manager uses only change feeds from local shards, and not going though the fabric layer, it is interesting to investigate switching to "normal" mode. Details: Previous couch_replicator's change feeds go through the fabric layer, so longpoll is needed because the set of shards which signaled the update, might not be the shards change feed will hit first when they are spawned. In that case if "normal" mode is used, then the change feed could immediately return without waiting for the new change to propagate via internal replication. Longpoll mode ensures there is a wait for shards to receive the change before stopping. CouchDB 2.0 replicator manager uses change feeds on individual shards. It seems even though "longpoll" mode will still work correctly (replicator manager is notified there is an update to the database, then gets changes from last checkpoint, and then saves the new checkpoint), "normal" mode might be more semantically correct, as effectively "longpoll" is used as normal – it never waits on changes.
      Hide
      kxepal Alexander Shorin added a comment -

      I thought about all of this for a while and have another question: how "longpoll" is different from continuous feed with non-infinity timeout? Now "longpoll" inherited this option, but it makes no sense for it.

      The question about "normal" vs "longpoll" is about "Do we want to keep a connection listening for the changes, or spawn it eventually all the time to ask if something had changed". I think it's quite strange to fallback into HTTP 1.0 times today.

      Show
      kxepal Alexander Shorin added a comment - I thought about all of this for a while and have another question: how "longpoll" is different from continuous feed with non-infinity timeout? Now "longpoll" inherited this option, but it makes no sense for it. The question about "normal" vs "longpoll" is about "Do we want to keep a connection listening for the changes, or spawn it eventually all the time to ask if something had changed". I think it's quite strange to fallback into HTTP 1.0 times today.
      Hide
      vatamane Nick Vatamaniuc added a comment -

      Good questions Alex.

      "longpoll" behavior, the way I understand it, would wait for a change to happen (however long the timeout is, if 'infinity' then forever), and return when it sees a change. The key part is the since parameter though. So if current sequence id is 1 and we launch a "longpoll" within since=1&timeout=infinity, and then never make any changes to the database, it would wait forever. But if we give it since=0, it will return immediately with all the changes it knows since sequence=0, in this case 1 change and then exit.

      However, in the replicator manager case, we have a db event listener. {{ https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_manager.erl#L391 }}. It notifies first if there is a change, and only then the change feeds are launched. So it is guaranteed there'd be some changes and the feeds would exit when those changes are processed. In practice this means the feeds act like "normal" feeds. It seems they'd never take advantage of the "waiting till a change happens" part of the "longpoll" mode. Another way to look at it is, in the idle state, the couch replicator manager would not have any active change feeds running. They are spawned only when event listener sees databases change and then exit.

      That is why I thought, even though the behavior might not be different in practice here between longpoll and normal. Making it "normal" makes more sense, as pretty much acts like "normal" anyway. But again, I am not too familiar with the code, and there are many subtle nuances in that code. So wanted to bring it up for a wider discussion.

      (Note: there other possible pattern is to use continuous feeds and then unlink and kill them on rescan. So in this case, in the idle state, there would active change feed processes alive and waiting on each replicator databases. But that changes the logic a bit more, which might introduce new subtle bugs).

      Show
      vatamane Nick Vatamaniuc added a comment - Good questions Alex. "longpoll" behavior, the way I understand it, would wait for a change to happen (however long the timeout is, if 'infinity' then forever), and return when it sees a change. The key part is the since parameter though. So if current sequence id is 1 and we launch a "longpoll" within since=1&timeout=infinity, and then never make any changes to the database, it would wait forever. But if we give it since=0, it will return immediately with all the changes it knows since sequence=0, in this case 1 change and then exit. However, in the replicator manager case, we have a db event listener. {{ https://github.com/apache/couchdb-couch-replicator/blob/master/src/couch_replicator_manager.erl#L391 }}. It notifies first if there is a change, and only then the change feeds are launched. So it is guaranteed there'd be some changes and the feeds would exit when those changes are processed. In practice this means the feeds act like "normal" feeds. It seems they'd never take advantage of the "waiting till a change happens" part of the "longpoll" mode. Another way to look at it is, in the idle state, the couch replicator manager would not have any active change feeds running. They are spawned only when event listener sees databases change and then exit. That is why I thought, even though the behavior might not be different in practice here between longpoll and normal. Making it "normal" makes more sense, as pretty much acts like "normal" anyway. But again, I am not too familiar with the code, and there are many subtle nuances in that code. So wanted to bring it up for a wider discussion. (Note: there other possible pattern is to use continuous feeds and then unlink and kill them on rescan. So in this case, in the idle state, there would active change feed processes alive and waiting on each replicator databases. But that changes the logic a bit more, which might introduce new subtle bugs).
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit d00b981445c03622497088eb872059ab4f48b298 in couchdb-couch-replicator's branch refs/heads/master from Nick Vatamaniuc
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=d00b981 ]

      Prevent replicator manager change feeds from getting stuck

      Switch them them from `longpoll` to `normal`

      This would prevent them being stuck. That could happen if more than one
      `resume_scan` message arrives for the same shard. The first time a longpoll
      changef feed would finish and end sequence is checkpointed. But if another
      resume_scan arrives and database hasn't changed then the longpoll change
      feed would hang until db is updated.

      The reason there would be multiple `resume_scan` messages is because there
      is a race condition between db update handler and scanner component. They are
      both started asynchronously roughly at the same. Scanner finds new shard while
      db handler notices changes for those shards. If shards are modified quickly
      after they are discovered by the scanner both of those components would issue
      a resume_scan.

      The effect of this would be more pronounced if there are a large number of
      _replicator shards and constant db creation/deletion/updates.

      COUCHDB-2964

      Show
      jira-bot ASF subversion and git services added a comment - Commit d00b981445c03622497088eb872059ab4f48b298 in couchdb-couch-replicator's branch refs/heads/master from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=d00b981 ] Prevent replicator manager change feeds from getting stuck Switch them them from `longpoll` to `normal` This would prevent them being stuck. That could happen if more than one `resume_scan` message arrives for the same shard. The first time a longpoll changef feed would finish and end sequence is checkpointed. But if another resume_scan arrives and database hasn't changed then the longpoll change feed would hang until db is updated. The reason there would be multiple `resume_scan` messages is because there is a race condition between db update handler and scanner component. They are both started asynchronously roughly at the same. Scanner finds new shard while db handler notices changes for those shards. If shards are modified quickly after they are discovered by the scanner both of those components would issue a resume_scan. The effect of this would be more pronounced if there are a large number of _replicator shards and constant db creation/deletion/updates. COUCHDB-2964
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit d00b981445c03622497088eb872059ab4f48b298 in couchdb-couch-replicator's branch refs/heads/63012-scheduler from Nick Vatamaniuc
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=d00b981 ]

      Prevent replicator manager change feeds from getting stuck

      Switch them them from `longpoll` to `normal`

      This would prevent them being stuck. That could happen if more than one
      `resume_scan` message arrives for the same shard. The first time a longpoll
      changef feed would finish and end sequence is checkpointed. But if another
      resume_scan arrives and database hasn't changed then the longpoll change
      feed would hang until db is updated.

      The reason there would be multiple `resume_scan` messages is because there
      is a race condition between db update handler and scanner component. They are
      both started asynchronously roughly at the same. Scanner finds new shard while
      db handler notices changes for those shards. If shards are modified quickly
      after they are discovered by the scanner both of those components would issue
      a resume_scan.

      The effect of this would be more pronounced if there are a large number of
      _replicator shards and constant db creation/deletion/updates.

      COUCHDB-2964

      Show
      jira-bot ASF subversion and git services added a comment - Commit d00b981445c03622497088eb872059ab4f48b298 in couchdb-couch-replicator's branch refs/heads/63012-scheduler from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=d00b981 ] Prevent replicator manager change feeds from getting stuck Switch them them from `longpoll` to `normal` This would prevent them being stuck. That could happen if more than one `resume_scan` message arrives for the same shard. The first time a longpoll changef feed would finish and end sequence is checkpointed. But if another resume_scan arrives and database hasn't changed then the longpoll change feed would hang until db is updated. The reason there would be multiple `resume_scan` messages is because there is a race condition between db update handler and scanner component. They are both started asynchronously roughly at the same. Scanner finds new shard while db handler notices changes for those shards. If shards are modified quickly after they are discovered by the scanner both of those components would issue a resume_scan. The effect of this would be more pronounced if there are a large number of _replicator shards and constant db creation/deletion/updates. COUCHDB-2964
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit d00b981445c03622497088eb872059ab4f48b298 in couchdb-couch-replicator's branch refs/heads/COUCHDB-3288-remove-public-db-record from Nick Vatamaniuc
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=d00b981 ]

      Prevent replicator manager change feeds from getting stuck

      Switch them them from `longpoll` to `normal`

      This would prevent them being stuck. That could happen if more than one
      `resume_scan` message arrives for the same shard. The first time a longpoll
      changef feed would finish and end sequence is checkpointed. But if another
      resume_scan arrives and database hasn't changed then the longpoll change
      feed would hang until db is updated.

      The reason there would be multiple `resume_scan` messages is because there
      is a race condition between db update handler and scanner component. They are
      both started asynchronously roughly at the same. Scanner finds new shard while
      db handler notices changes for those shards. If shards are modified quickly
      after they are discovered by the scanner both of those components would issue
      a resume_scan.

      The effect of this would be more pronounced if there are a large number of
      _replicator shards and constant db creation/deletion/updates.

      COUCHDB-2964

      Show
      jira-bot ASF subversion and git services added a comment - Commit d00b981445c03622497088eb872059ab4f48b298 in couchdb-couch-replicator's branch refs/heads/ COUCHDB-3288 -remove-public-db-record from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=d00b981 ] Prevent replicator manager change feeds from getting stuck Switch them them from `longpoll` to `normal` This would prevent them being stuck. That could happen if more than one `resume_scan` message arrives for the same shard. The first time a longpoll changef feed would finish and end sequence is checkpointed. But if another resume_scan arrives and database hasn't changed then the longpoll change feed would hang until db is updated. The reason there would be multiple `resume_scan` messages is because there is a race condition between db update handler and scanner component. They are both started asynchronously roughly at the same. Scanner finds new shard while db handler notices changes for those shards. If shards are modified quickly after they are discovered by the scanner both of those components would issue a resume_scan. The effect of this would be more pronounced if there are a large number of _replicator shards and constant db creation/deletion/updates. COUCHDB-2964
      Hide
      jira-bot ASF subversion and git services added a comment -

      Commit d00b981445c03622497088eb872059ab4f48b298 in couchdb-couch-replicator's branch refs/heads/COUCHDB-3287-pluggable-storage-engines from Nick Vatamaniuc
      [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=d00b981 ]

      Prevent replicator manager change feeds from getting stuck

      Switch them them from `longpoll` to `normal`

      This would prevent them being stuck. That could happen if more than one
      `resume_scan` message arrives for the same shard. The first time a longpoll
      changef feed would finish and end sequence is checkpointed. But if another
      resume_scan arrives and database hasn't changed then the longpoll change
      feed would hang until db is updated.

      The reason there would be multiple `resume_scan` messages is because there
      is a race condition between db update handler and scanner component. They are
      both started asynchronously roughly at the same. Scanner finds new shard while
      db handler notices changes for those shards. If shards are modified quickly
      after they are discovered by the scanner both of those components would issue
      a resume_scan.

      The effect of this would be more pronounced if there are a large number of
      _replicator shards and constant db creation/deletion/updates.

      COUCHDB-2964

      Show
      jira-bot ASF subversion and git services added a comment - Commit d00b981445c03622497088eb872059ab4f48b298 in couchdb-couch-replicator's branch refs/heads/ COUCHDB-3287 -pluggable-storage-engines from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-replicator.git;h=d00b981 ] Prevent replicator manager change feeds from getting stuck Switch them them from `longpoll` to `normal` This would prevent them being stuck. That could happen if more than one `resume_scan` message arrives for the same shard. The first time a longpoll changef feed would finish and end sequence is checkpointed. But if another resume_scan arrives and database hasn't changed then the longpoll change feed would hang until db is updated. The reason there would be multiple `resume_scan` messages is because there is a race condition between db update handler and scanner component. They are both started asynchronously roughly at the same. Scanner finds new shard while db handler notices changes for those shards. If shards are modified quickly after they are discovered by the scanner both of those components would issue a resume_scan. The effect of this would be more pronounced if there are a large number of _replicator shards and constant db creation/deletion/updates. COUCHDB-2964

        People

        • Assignee:
          kzx kzx
          Reporter:
          vatamane Nick Vatamaniuc
        • Votes:
          0 Vote for this issue
          Watchers:
          3 Start watching this issue

          Dates

          • Created:
            Updated:

            Development