CouchDB
  1. CouchDB
  2. COUCHDB-1199

Doc in _replicator db missing when using filter property

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.1
    • Fix Version/s: 1.1.1, 1.2
    • Component/s: Replication
    • Labels:
      None
    • Environment:

      Ubuntu 10.10

      Description

      Scenario1 below works as expected, Scenario2 fails.

      Scenario1
      curl -X POST 'http://127.0.0.1:5984/_replicator/' -H "Content-Type: application/json" -d {\"_id\":\"test1_to_test2\"\,\"source\":\"my_culture\"\,\"target\":\"http:\/\/127.0.0.1:5984\/test2\"\,\"create_target\":\true}

      {"ok":true,"id":"test1_to_test2","rev":"1-df297eda4880633bc0442590724014ff"}

      Doc is created in _replicator
      Replication completes successfully

      Scenario2
      curl -X POST 'http://127.0.0.1:5984/_replicator/' -H "Content-Type: application/json" -d

      {\"_id\":\"test1_to_test2\"\,\"source\":\"my_culture\"\,\"target\":\"http:\/\/127.0.0.1:5984\/test2\"\,\"create_target\":\true\,\"filter\":\"http:\/\/127.0.0.1:5984\/my_culture\/profile\/app_ddocs\"} {"ok":true,"id":"test1_to_test2","rev":"1-97641b372d500d842688d217f97081da"}

      Doc is not created in _replicator (in spite of 'ok' response above)
      No replication occurs.

      Now, I am not sure whether I got the right syntax in the filter property since I could find no documentation for it. In particular, whether the filter should be in the source db or _replicator. And do we use a full URL as above or just ddoc/filter name? The filter documented in Scenario2 does exist in source db.

      In any case, the doc in _replicator ought not to be getting lost as does happen.

        Activity

        Hide
        Filipe Manana added a comment -

        Fix applied to trunk and 1.1.x.
        Thanks Andrew

        Show
        Filipe Manana added a comment - Fix applied to trunk and 1.1.x. Thanks Andrew
        Hide
        Filipe Manana added a comment -

        Andrew, the attached patch fixes the issue (for branch 1.1.x).

        Will commit it soon.

        Show
        Filipe Manana added a comment - Andrew, the attached patch fixes the issue (for branch 1.1.x). Will commit it soon.
        Hide
        Filipe Manana added a comment -

        Thanks Andrew.

        I was able to reproduce it. It happens only when there's a filter specified which doesn't exist, throwing an exception and causing the replicator database gen_server to be restarted 10 times by the supervisor, after which is no longer restarted anymore.

        I'll fix this asap.

        Show
        Filipe Manana added a comment - Thanks Andrew. I was able to reproduce it. It happens only when there's a filter specified which doesn't exist, throwing an exception and causing the replicator database gen_server to be restarted 10 times by the supervisor, after which is no longer restarted anymore. I'll fix this asap.
        Hide
        andrew henderson added a comment -

        Filipe, thank you for the helpful suggestions.
        It gets worse before getting better. Upon following your advice regarding -d '

        {...}

        ' and inserting the same doc with INCORRECT syntax for filter as in scenario2 above, the couchdb server now crashes immediately. I have about two seconds after restarting the server to delete the doc before it crashes again.
        curl -X POST 'http://127.0.0.1:5984/_replicator/' -H "Content-Type: application/json" -d '

        {"_id":"test1_to_test2","source":"my_culture","target":"http://127.0.0.1:5984/test2","create_target":true,"filter":"http://127.0.0.1:5984/my_culture/profile/app_ddocs"}

        '
        – CouchDB Crashes. Restart –
        curl -X GET http://127.0.0.1:5984/_replicator/_all_docs {"total_rows":2,"offset":0,"rows":[ {"id":"_design/_replicator","key":"_design/_replicator","value":{"rev":"1-5bfa2c99eefe2b2eb4962db50aa3cfd4"}}, {"id":"test1_to_test2","key":"test1_to_test2","value":{"rev":"1-3241d604304f6db88285b3696406a4dd"}}
        – CouchDB Crashes again –

        The good news is that using correct syntax to insert a _replicator with filter is safe and does replicate as expected.
        curl -X POST 'http://127.0.0.1:5984/_replicator/' -H "Content-Type: application/json" -d '

        {"_id":"test1_to_test2","source":"my_culture","target":"http://127.0.0.1:5984/test2","create_target":true,"filter":"profile/app_ddocs"}

        '

        {"ok":true,"id":"test1_to_test2","rev":"1-ebee8391fe78cb4155f7f134d76af77e"}

        admin >
        curl -X GET http://127.0.0.1:5984/_replicator/_all_docs{"total_rows":2,"offset":0,"rows":[ {"id":"_design/_replicator","key":"_design/_replicator","value":{"rev":"1-5bfa2c99eefe2b2eb4962db50aa3cfd4"}}, {"id":"test1_to_test2","key":"test1_to_test2","value":{"rev":"3-a004cf732565fecff16ec00fc9b7491d"}} ]}
        I would suggest downgrading the criticality of this bug, but fixing it anyway since it can be very problematic to mistakenly insert a _replicator filter doc using incorrect syntax.

        • Andrew
        Show
        andrew henderson added a comment - Filipe, thank you for the helpful suggestions. It gets worse before getting better. Upon following your advice regarding -d ' {...} ' and inserting the same doc with INCORRECT syntax for filter as in scenario2 above, the couchdb server now crashes immediately. I have about two seconds after restarting the server to delete the doc before it crashes again. curl -X POST 'http://127.0.0.1:5984/_replicator/' -H "Content-Type: application/json" -d ' {"_id":"test1_to_test2","source":"my_culture","target":"http://127.0.0.1:5984/test2","create_target":true,"filter":"http://127.0.0.1:5984/my_culture/profile/app_ddocs"} ' – CouchDB Crashes. Restart – curl -X GET http://127.0.0.1:5984/_replicator/_all_docs {"total_rows":2,"offset":0,"rows":[ {"id":"_design/_replicator","key":"_design/_replicator","value":{"rev":"1-5bfa2c99eefe2b2eb4962db50aa3cfd4"}}, {"id":"test1_to_test2","key":"test1_to_test2","value":{"rev":"1-3241d604304f6db88285b3696406a4dd"}} – CouchDB Crashes again – The good news is that using correct syntax to insert a _replicator with filter is safe and does replicate as expected. curl -X POST 'http://127.0.0.1:5984/_replicator/' -H "Content-Type: application/json" -d ' {"_id":"test1_to_test2","source":"my_culture","target":"http://127.0.0.1:5984/test2","create_target":true,"filter":"profile/app_ddocs"} ' {"ok":true,"id":"test1_to_test2","rev":"1-ebee8391fe78cb4155f7f134d76af77e"} admin > curl -X GET http://127.0.0.1:5984/_replicator/_all_docs {"total_rows":2,"offset":0,"rows":[ {"id":"_design/_replicator","key":"_design/_replicator","value":{"rev":"1-5bfa2c99eefe2b2eb4962db50aa3cfd4"}}, {"id":"test1_to_test2","key":"test1_to_test2","value":{"rev":"3-a004cf732565fecff16ec00fc9b7491d"}} ]} I would suggest downgrading the criticality of this bug, but fixing it anyway since it can be very problematic to mistakenly insert a _replicator filter doc using incorrect syntax. Andrew
        Hide
        Filipe Manana added a comment -

        Hi Andrew,

        The filter is always relative to the source and its format is like "ddocname/filtername", exactly like it is for _changes requests. I believe this is documented somewhere in the wiki (for _changes requests at least).

        One thing I find odd is that for scenario 2 you got the reply:

        {"ok":true,"id":"test1_to_test2","rev":"1-97641b372d500d842688d217f97081da"}

        But you don't see the doc in the _replicator database. This is odd since when a document is rejected, via the default doc validate update function in _design/_replicator document, you should get an explicit error message. For your case, the document seems perfectly fine for this validate update function.

        Can you query /_replicator/_all_docs and confirm the document is not there? Ideally you could paste it here.
        thanks

        As a side note, when using curl you can surround the argument to -d in single quotes, like this you avoid all that escaping which makes things harder to read. E.g. curl ... -d '

        {"source": "mysource", "target": "mytarget"}

        '

        Show
        Filipe Manana added a comment - Hi Andrew, The filter is always relative to the source and its format is like "ddocname/filtername", exactly like it is for _changes requests. I believe this is documented somewhere in the wiki (for _changes requests at least). One thing I find odd is that for scenario 2 you got the reply: {"ok":true,"id":"test1_to_test2","rev":"1-97641b372d500d842688d217f97081da"} But you don't see the doc in the _replicator database. This is odd since when a document is rejected, via the default doc validate update function in _design/_replicator document, you should get an explicit error message. For your case, the document seems perfectly fine for this validate update function. Can you query /_replicator/_all_docs and confirm the document is not there? Ideally you could paste it here. thanks As a side note, when using curl you can surround the argument to -d in single quotes, like this you avoid all that escaping which makes things harder to read. E.g. curl ... -d ' {"source": "mysource", "target": "mytarget"} '

          People

          • Assignee:
            Filipe Manana
            Reporter:
            andrew henderson
          • Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development