CouchDB
  1. CouchDB
  2. COUCHDB-885

Attachments introduce conflicts after replication.

    Details

    • Type: Bug Bug
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 1.0.1
    • Fix Version/s: 1.0.3, 1.1, 1.2
    • Component/s: Database Core
    • Labels:
      None
    • Environment:

      Mac OSX, Windows XP, Windows 7

    • Skill Level:
      Committers Level (Medium to Hard)

      Description

      Step to reproduce the bug:

      1. Make database "test" on a remote couchdb server that reside on a different machine!
      2. Create new document: "http://remote-server:5984/test/doc"
      3. Create database "replica" on the local couchdb server.
      4. Trigger pull replication http://remote-server:5984/test -> http://localhost:5984/replica
      5. Attach a file to the replicated document on the local couchdb server.
      6. Trigger push replication http://localhost:5984/replica -> http://remote-server:5984/test

      The document in the test database shows one conflict. If instead attachment the document is updated with a new field after the replication no conflict.
      The same sequence shows no conflicts on CouchDB 0.11.

                            • test.sh ****************

      #!/usr/bin/env bash

      LOCAL="http://localhost:5984"
      REMOTE="http://koleto.couchone.com"

      echo "====================================================="
      echo "This sequence create a conflict after the replication"
      echo "====================================================="

      echo "Cleanup"
      curl --silent -X DELETE $LOCAL/replica
      curl --silent -X DELETE $REMOTE/test

      echo "Create database:" $REMOTE/test
      curl --silent -X PUT $REMOTE/test

      echo "Create empty document:"
      curl --silent -d '{}' -X PUT $REMOTE/test/doc

      echo "Create local databse:" $LOCAL/replica
      curl --silent -X PUT $LOCAL/replica

      echo "Pull replication:" REMOTE/test "->" $LOCAL/replica
      curl --silent -X POST -d "

      {\"source\":\"$REMOTE/test\",\"target\":\"$LOCAL/replica\"}

      " -H 'Content-Type: application/json' $LOCAL/_replicate

      echo "Put an attachment to the document:"
      echo "foobar!" | curl --silent --upload-file - -X PUT "$LOCAL/replica/doc/foo.txt?rev=1-967a00dff5e02add41819138abb3284d"

      echo "Push replication:" $LOCAL/replica "->" $REMOTE/test
      curl --silent -X POST -d "

      {\"source\":\"$LOCAL/replica\",\"target\":\"$REMOTE/test\"}

      " -H 'Content-Type: application/json' $LOCAL/_replicate

      echo '############## Get one conflic #####################'
      curl --silent -X GET $REMOTE/test/doc?conflicts=true

      echo "====================================================="
      echo "The same sequence with adding a field to the document"
      echo "====================================================="

      echo "Cleanup"
      curl --silent -X DELETE $LOCAL/replica
      curl --silent -X DELETE $REMOTE/test

      echo "Create database:" $REMOTE/test
      curl --silent -X PUT $REMOTE/test

      echo "Create empty document:"
      curl --silent -d '{}' -X PUT $REMOTE/test/doc

      echo "Create local databse:" $LOCAL/replica
      curl --silent -X PUT $LOCAL/replica

      echo "Pull replication:" REMOTE/test "->" $LOCAL/replica
      curl --silent -X POST -d "

      {\"source\":\"$REMOTE/test\",\"target\":\"$LOCAL/replica\"}

      " -H 'Content-Type: application/json' $LOCAL/_replicate

      echo "Adding a new field:"
      curl --silent -d '

      {"foo":"bar" , "_rev" : "1-967a00dff5e02add41819138abb3284d" }

      ' -X PUT $LOCAL/replica/doc?rev=1-967a00dff5e02add41819138abb3284d

      echo "Push replication:" $LOCAL/replica "->" $REMOTE/test
      curl --silent -X POST -d "

      {\"source\":\"$LOCAL/replica\",\"target\":\"$REMOTE/test\"}

      " -H 'Content-Type: application/json' $LOCAL/_replicate

      echo '###### No conflicts after the replication ########'
      curl --silent -X GET $REMOTE/test/doc?conflicts=true

      1. COUCHDB-885.sh
        0.8 kB
        Robert Newson

        Issue Links

          Activity

          Hide
          Sebastian Cohnen added a comment -

          I cannot reproduce.

          But I'm not sure if I understand your steps correctly, so I've written a bash script (see http://www.friendpaste.com/6SrHCU1lseUURuwTJEpCpk) to execute your steps. Could you have a look and verify that I understood your steps correctly?

          Show
          Sebastian Cohnen added a comment - I cannot reproduce. But I'm not sure if I understand your steps correctly, so I've written a bash script (see http://www.friendpaste.com/6SrHCU1lseUURuwTJEpCpk ) to execute your steps. Could you have a look and verify that I understood your steps correctly?
          Hide
          Sebastian Cohnen added a comment -

          okay, got an error in my script (I've updated the paste). Now I can confirm the behavior.

          Show
          Sebastian Cohnen added a comment - okay, got an error in my script (I've updated the paste). Now I can confirm the behavior.
          Hide
          Klaus Trainer added a comment -

          There is an easier way (only three steps) to get something like that. It doesn't necessarily involve attachments and replication.

          curl -X PUT http://127.0.0.1:5984/foobase/doc -d '

          {"foo":"bar"}

          curl -X POST http://127.0.0.1:5984/foobase/_bulk_docs -H 'Content-Type: application/json' -d '{"all_or_nothing": true, "docs":[

          {"_id": "doc", "_rev":"2-0e871ef78849b0c206091f1a7af6ec41", "foo":"bar"}

          ]}'

          curl -X DELETE http://127.0.0.1:5984/foobase/doc?rev=3-b06fcd1c1c9e0ec7c480ee8aa467bf3b

          Show
          Klaus Trainer added a comment - There is an easier way (only three steps) to get something like that. It doesn't necessarily involve attachments and replication. curl -X PUT http://127.0.0.1:5984/foobase/doc -d ' {"foo":"bar"} curl -X POST http://127.0.0.1:5984/foobase/_bulk_docs -H 'Content-Type: application/json' -d '{"all_or_nothing": true, "docs":[ {"_id": "doc", "_rev":"2-0e871ef78849b0c206091f1a7af6ec41", "foo":"bar"} ]}' curl -X DELETE http://127.0.0.1:5984/foobase/doc?rev=3-b06fcd1c1c9e0ec7c480ee8aa467bf3b
          Hide
          Klaus Trainer added a comment -

          Note that the procedure I've previously described will create a conflict:

          curl -X GET http://127.0.0.1:5984/foobase/doc?conflicts=true

          {"_id":"doc","_rev":"3-b06fcd1c1c9e0ec7c480ee8aa467bf3b","foo":"bar","_conflicts":["1-4c6114c65e295552ab1019e2b046b10e"]}
          Show
          Klaus Trainer added a comment - Note that the procedure I've previously described will create a conflict: curl -X GET http://127.0.0.1:5984/foobase/doc?conflicts=true {"_id":"doc","_rev":"3-b06fcd1c1c9e0ec7c480ee8aa467bf3b","foo":"bar","_conflicts":["1-4c6114c65e295552ab1019e2b046b10e"]}
          Hide
          Jan Lehnardt added a comment -

          At least Klaus' latest steps look like intended behaviour to me. What am I missing?

          Show
          Jan Lehnardt added a comment - At least Klaus' latest steps look like intended behaviour to me. What am I missing?
          Hide
          Klaus Trainer added a comment -

          Just read Nikolai's description again. My previous two comments here were maybe beside the point.

          I think that this issue might be related to COUCHDB-902, which is not fixed yet.

          The problem in COUCHDB-902 is that a conflict occurs when it shouldn't. I left COUCHDB-902 with a patch for review. Now, I just discovered Filipe's latest comment (on COUCHDB-902) that I've obviously missed. I find it interesting that what Filipe's saying contradicts what I've been observing. Anyway, I'll get back to COUCHDB-902 and check again, as soon as I get some time for that. It should definitely get fixed.

          I suggest that we leave this issue open until COUCHDB-902 is fixed, and get back to it then to see if it's still an issue.

          Show
          Klaus Trainer added a comment - Just read Nikolai's description again. My previous two comments here were maybe beside the point. I think that this issue might be related to COUCHDB-902 , which is not fixed yet. The problem in COUCHDB-902 is that a conflict occurs when it shouldn't. I left COUCHDB-902 with a patch for review. Now, I just discovered Filipe's latest comment (on COUCHDB-902 ) that I've obviously missed. I find it interesting that what Filipe's saying contradicts what I've been observing. Anyway, I'll get back to COUCHDB-902 and check again, as soon as I get some time for that. It should definitely get fixed. I suggest that we leave this issue open until COUCHDB-902 is fixed, and get back to it then to see if it's still an issue.
          Hide
          Nikolai Teofilov added a comment -

          I also think COUCHDB-902 and COUCHDB-885 are related!
          Somewhere in the dev list i read about problem with replication and not released file descriptors.

          Show
          Nikolai Teofilov added a comment - I also think COUCHDB-902 and COUCHDB-885 are related! Somewhere in the dev list i read about problem with replication and not released file descriptors.
          Hide
          Klaus Trainer added a comment -

          I've scrutinized the issue again. As expected, it has been resolved with COUCHDB-902. Hence, I'd propose that somebody updates this issue to have the same status like COUCHDB-902.

          Show
          Klaus Trainer added a comment - I've scrutinized the issue again. As expected, it has been resolved with COUCHDB-902 . Hence, I'd propose that somebody updates this issue to have the same status like COUCHDB-902 .
          Hide
          Paul Joseph Davis added a comment -

          Fixed by COUCHDB-902

          Show
          Paul Joseph Davis added a comment - Fixed by COUCHDB-902
          Hide
          James Howe added a comment - - edited

          This issue is still present in 1.0.2, even though COUCHDB-902 is fixed.
          Specifically the following process creates a conflict:

          couch1 - add new document A in test_db
          replicate

          {source: test_db, target: http://couch2/test_db}

          couch1 - add attachment to A
          replicate

          {source: test_db, target: http://couch2/test_db}

          couch2/test_db/A?open_revs=all now shows a conflict

          Show
          James Howe added a comment - - edited This issue is still present in 1.0.2, even though COUCHDB-902 is fixed. Specifically the following process creates a conflict: couch1 - add new document A in test_db replicate {source: test_db, target: http://couch2/test_db} couch1 - add attachment to A replicate {source: test_db, target: http://couch2/test_db} couch2/test_db/A?open_revs=all now shows a conflict
          Hide
          Filipe Manana added a comment -

          Hi!

          I think you're confusing a multiple revisions with conflicting revisions.
          For that particular test, do both revisions start with the same position (what comes before the "-")?
          If they do, you have a conflict, otherwise you don't.

          Also, do a GET couch2/test_db/A?conflicts=true, this is the best way to see if you have conflicts (look for a property named "conflicts" which should have an array value if there are conflicts)

          Show
          Filipe Manana added a comment - Hi! I think you're confusing a multiple revisions with conflicting revisions. For that particular test, do both revisions start with the same position (what comes before the "-")? If they do, you have a conflict, otherwise you don't. Also, do a GET couch2/test_db/A?conflicts=true, this is the best way to see if you have conflicts (look for a property named "conflicts" which should have an array value if there are conflicts)
          Hide
          James Howe added a comment - - edited

          If pull replication is performed -

          {source: http://couch1/test_db, target: test_db}

          - then the open_revs list shows the expected one open branch, not two.

          Retracted:
          And I think you're confusing deleted conflicts with multiple revisions.
          As expected, the _conflicts array is not there, as the conflict is created in a deleted state.

          Show
          James Howe added a comment - - edited If pull replication is performed - {source: http://couch1/test_db, target: test_db} - then the open_revs list shows the expected one open branch, not two. Retracted: And I think you're confusing deleted conflicts with multiple revisions. As expected, the _conflicts array is not there, as the conflict is created in a deleted state.
          Hide
          Filipe Manana added a comment -

          I don't think so. I don't see any _deleted_conflicts when querying with ?deleted_conflicts=true. I'm referring to the following test case:

          couch1 - add new document A in test_db
          replicate

          {source: test_db, target: http://couch2/test_db}


          couch1 - add attachment to A
          replicate

          {source: test_db, target: http://couch2/test_db}


          couch2/test_db/A?open_revs=all now shows a conflict

          I was able to reproduce it with a push replication in 1.0.2 only. It doesn't happen with trunk however.
          So just ignore it.

          Show
          Filipe Manana added a comment - I don't think so. I don't see any _deleted_conflicts when querying with ?deleted_conflicts=true. I'm referring to the following test case: couch1 - add new document A in test_db replicate {source: test_db, target: http://couch2/test_db} couch1 - add attachment to A replicate {source: test_db, target: http://couch2/test_db} couch2/test_db/A?open_revs=all now shows a conflict I was able to reproduce it with a push replication in 1.0.2 only. It doesn't happen with trunk however. So just ignore it.
          Hide
          James Howe added a comment - - edited

          Sorry, there is a _conflicts, it's not deleted at all. Having some JSON-blindness obviously.

          GET http://couch2/test_db/6028bdb0ea5f748321b3ef1e980ef0a5?open_revs=all
          [
          {
          "ok": {
          "_id": "6028bdb0ea5f748321b3ef1e980ef0a5"
          "_rev": "2-49be403b7a6ddb504b657a86932df567"
          "_attachments": {
          "text_attachement":

          { "content_type": "application/octet-stream" "revpos": 2 "length": 7 "stub": true }

          }
          }
          }
          {
          "ok":

          { "_id": "6028bdb0ea5f748321b3ef1e980ef0a5" "_rev": "1-967a00dff5e02add41819138abb3284d" }

          }
          ]

          GET http://couch2/test_db/6028bdb0ea5f748321b3ef1e980ef0a5?conflicts=true
          {
          "_id": "6028bdb0ea5f748321b3ef1e980ef0a5"
          "_rev": "2-49be403b7a6ddb504b657a86932df567"
          "_conflicts": [
          "1-967a00dff5e02add41819138abb3284d"
          ]
          "_attachments": {
          "text_attachement":

          { "content_type": "application/octet-stream" "revpos": 2 "length": 7 "stub": true }

          }
          }

          If this were caused by COUCHDB-902 then either both or neither should be fixed in 1.0.2 - this is my concern.

          Show
          James Howe added a comment - - edited Sorry, there is a _conflicts, it's not deleted at all. Having some JSON-blindness obviously. GET http://couch2/test_db/6028bdb0ea5f748321b3ef1e980ef0a5?open_revs=all [ { "ok": { "_id": "6028bdb0ea5f748321b3ef1e980ef0a5" "_rev": "2-49be403b7a6ddb504b657a86932df567" "_attachments": { "text_attachement": { "content_type": "application/octet-stream" "revpos": 2 "length": 7 "stub": true } } } } { "ok": { "_id": "6028bdb0ea5f748321b3ef1e980ef0a5" "_rev": "1-967a00dff5e02add41819138abb3284d" } } ] GET http://couch2/test_db/6028bdb0ea5f748321b3ef1e980ef0a5?conflicts=true { "_id": "6028bdb0ea5f748321b3ef1e980ef0a5" "_rev": "2-49be403b7a6ddb504b657a86932df567" "_conflicts": [ "1-967a00dff5e02add41819138abb3284d" ] "_attachments": { "text_attachement": { "content_type": "application/octet-stream" "revpos": 2 "length": 7 "stub": true } } } If this were caused by COUCHDB-902 then either both or neither should be fixed in 1.0.2 - this is my concern.
          Hide
          Robert Newson added a comment -

          A script that reproduces the conflict on the target against 1.0.2

          Note: You only need one instance of couchdb to reproduce the bug.

          Show
          Robert Newson added a comment - A script that reproduces the conflict on the target against 1.0.2 Note: You only need one instance of couchdb to reproduce the bug.
          Hide
          Clare Walsh added a comment -

          Have you tried that script against trunk?
          Running trunk I can't reproduce the error using push, pull, local, remote, single or multiple instances, or combos of those as described above.
          (Haven't run the script as the steps seem to be exactly as Filipe described).

          Though we can reproduce the error on 1.0.2 using almost any of the sets of described steps so the affect versions should definitely be updated to reflect this.

          Show
          Clare Walsh added a comment - Have you tried that script against trunk? Running trunk I can't reproduce the error using push, pull, local, remote, single or multiple instances, or combos of those as described above. (Haven't run the script as the steps seem to be exactly as Filipe described). Though we can reproduce the error on 1.0.2 using almost any of the sets of described steps so the affect versions should definitely be updated to reflect this.
          Hide
          Robert Newson added a comment -

          hi Clare,

          Yes, I've verified that trunk works (this bug is not present).

          Filipe has a fix which I'm porting to 1.1.x and testing, Filipe has already done the same for 1.0.x.

          Show
          Robert Newson added a comment - hi Clare, Yes, I've verified that trunk works (this bug is not present). Filipe has a fix which I'm porting to 1.1.x and testing, Filipe has already done the same for 1.0.x.
          Hide
          Robert Newson added a comment -

          I should be clearer: the bug was introduced in the 0.11 release, preserved through 1.0, and finally squashed by the new replicator on trunk.

          Show
          Robert Newson added a comment - I should be clearer: the bug was introduced in the 0.11 release, preserved through 1.0, and finally squashed by the new replicator on trunk.
          Hide
          Robert Newson added a comment -

          The new fix will appear in 1.0.3 and 1.1.0.

          Show
          Robert Newson added a comment - The new fix will appear in 1.0.3 and 1.1.0.

            People

            • Assignee:
              Unassigned
              Reporter:
              Nikolai Teofilov
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development