CouchDB
  1. CouchDB
  2. COUCHDB-1363

callback invocation for docs added during couch_changes startup can be delayed by race condition

    Details

    • Type: Bug Bug
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: 1.0.3, 1.1.1
    • Fix Version/s: 1.2, 1.2.1, 1.3
    • Component/s: Database Core
    • Labels:
      None

      Description

      After subscribing to notifications it's necessary to re-open the #db a so that the header points at all updates for which the updater notifier has already fired events. In practice, this is rarely problematic because the next change will cause everything to catch up, but if a quick burst of changes happens while, e.g., replication is starting the replication can go stale. Detected by intermittent replicator_db js test failures.

        Activity

        Hide
        Randall Leeds added a comment -

        Fixed.

        To https://git-wip-us.apache.org/repos/asf/couchdb.git
        9c377a1..e82a0c9 1.0.x -> 1.0.x
        c9b20f2..6a04e33 1.1.x -> 1.1.x
        7f9376c..1bbe619 1.2.x -> 1.2.x
        2d90a12..573a7bb master -> master

        Show
        Randall Leeds added a comment - Fixed. To https://git-wip-us.apache.org/repos/asf/couchdb.git 9c377a1..e82a0c9 1.0.x -> 1.0.x c9b20f2..6a04e33 1.1.x -> 1.1.x 7f9376c..1bbe619 1.2.x -> 1.2.x 2d90a12..573a7bb master -> master
        Hide
        Filipe Manana added a comment -

        Go ahead Randall, it's a genuine issue. +1

        Show
        Filipe Manana added a comment - Go ahead Randall, it's a genuine issue. +1
        Hide
        Randall Leeds added a comment -

        Thanks for the review. I'll explain more (I was distracted by talking after the CouchBase meetup).

        I've run into this during the course of my work for COUCHDB-1350. As part of that, I had to change the replicator_db test and make the continuous_replication_survives_restart use bare db names instead of urls (the url changes after /_restart). I'm guessing I uncovered this race because the timing changes as a result of this different path.

        In any case, unless I'm missing something fundamental it's most definitely a bug and this patch definitely fixes it. I tested by removing the line to stop CouchDB after the tests, so I could leave the replication running after the failure, which was line 700 of replicator_db.js. The replication stays open in active tasks but the fourth change, to doc 'foo1000', isn't replicated until I PUT another change to the source database. That is because the Db handle that's passed into handle_changes is pointing at an old header and the update_notifier hasn't been started yet. The patch re-opens the db after subscribing to notifications, which ensures we can't miss an update and stall in this way.

        It probably doesn't bite many people, though (replication starts again as soon as the next change comes), but I can think of situations I've seen where this could be a problem.

        Thanks for catching the shadowing issue. I didn't notice the warning when I compiled. I thought that the DbName in the fun header would be matched against the DbName in the handle_changes header instead of shadowing. My intention was not to change that functionality and I should probably have left it alone.

        Let me know if it's okay to commit (- the shadowing). I don't see a better fix. Do you?

        Show
        Randall Leeds added a comment - Thanks for the review. I'll explain more (I was distracted by talking after the CouchBase meetup). I've run into this during the course of my work for COUCHDB-1350 . As part of that, I had to change the replicator_db test and make the continuous_replication_survives_restart use bare db names instead of urls (the url changes after /_restart). I'm guessing I uncovered this race because the timing changes as a result of this different path. In any case, unless I'm missing something fundamental it's most definitely a bug and this patch definitely fixes it. I tested by removing the line to stop CouchDB after the tests, so I could leave the replication running after the failure, which was line 700 of replicator_db.js. The replication stays open in active tasks but the fourth change, to doc 'foo1000', isn't replicated until I PUT another change to the source database. That is because the Db handle that's passed into handle_changes is pointing at an old header and the update_notifier hasn't been started yet. The patch re-opens the db after subscribing to notifications, which ensures we can't miss an update and stall in this way. It probably doesn't bite many people, though (replication starts again as soon as the next change comes), but I can think of situations I've seen where this could be a problem. Thanks for catching the shadowing issue. I didn't notice the warning when I compiled. I thought that the DbName in the fun header would be matched against the DbName in the handle_changes header instead of shadowing. My intention was not to change that functionality and I should probably have left it alone. Let me know if it's okay to commit (- the shadowing). I don't see a better fix. Do you?
        Hide
        Filipe Manana added a comment -

        Hi Randall,

        This would only minimize the "problem", as right after re-opening the database some changes can happen.

        As for the replicator_db.js test, the only case I can see this happening is when triggering a non-continuous replication, immediately after add some docs to the source database and then assert the docs were written to the target. I think this would be more a problem of the test then anything else. I don't recall if there's any test function which does that in replicator_db.js. Is there any? Which one did you find?

        Also the following line you changed is a bit dangerous:

        • fun({_, DbName}) when DbName == Db#db.name ->
          + fun({_, DbName}) ->

        The DbName inside the fun is shadowing the DbName in the handle_changes clause. This means you'll accept updates for any database.
        The compiler should give you a warning about this.

        I would also prefer you update the commit's title because this is not replication specific, but rather couch_changes specific.

        I'm mostly convinced it's a test issue anyway.

        Show
        Filipe Manana added a comment - Hi Randall, This would only minimize the "problem", as right after re-opening the database some changes can happen. As for the replicator_db.js test, the only case I can see this happening is when triggering a non-continuous replication, immediately after add some docs to the source database and then assert the docs were written to the target. I think this would be more a problem of the test then anything else. I don't recall if there's any test function which does that in replicator_db.js. Is there any? Which one did you find? Also the following line you changed is a bit dangerous: fun({_, DbName}) when DbName == Db#db.name -> + fun({_, DbName}) -> The DbName inside the fun is shadowing the DbName in the handle_changes clause. This means you'll accept updates for any database. The compiler should give you a warning about this. I would also prefer you update the commit's title because this is not replication specific, but rather couch_changes specific. I'm mostly convinced it's a test issue anyway.

          People

          • Assignee:
            Unassigned
            Reporter:
            Randall Leeds
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development