Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-3302

Attachment replication over low bandwidth network connections

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Replication
    • Labels:
      None

      Description

      Setup:

      Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit network connection (simulated locally with traffic shaping, see way below for an example).

      git clone https://github.com/apache/couchdb.git
      cd couchdb
      ./configure --disable-docs --disable-fauxton
      make release
      cd ..
      cp -r couchdb/rel/couchdb source
      cp -r couchdb/rel/couchdb target
      
      # set up local ini: chttpd / port: 5981 / 5983
      # set up vm.args: source@hostname.local / target@hostname.local
      
      # no admins
      
      Start both CouchDB in their own terminal windows: ./bin/couchdb
      
      # create all required databases, and our `t` test database
      curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
      
      # create 64MB attachments
      dd if=/dev/urandom of=att-64 bs=1024 count=65536
      
      # create doc on source
      curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: application/octet-stream' -d @att-64
      
      # replicate to target
      curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
      

      With the traffic shaping in place, the replication call doesn’t return, and eventually CouchDB fails with:

      [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator -------- Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
      {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
      
      [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0> -------- Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false" failed due to error {error,
          {'EXIT',
              {{{nocatch,{mp_parser_died,noproc}},
                [{couch_att,'-foldl/4-fun-0-',3,
                     [{file,"src/couch_att.erl"},{line,591}]},
                 {couch_att,fold_streamed_data,4,
                     [{file,"src/couch_att.erl"},{line,642}]},
                 {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
                 {couch_httpd_multipart,atts_to_mp,4,
                     [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
               {gen_server,call,
                   [<0.15778.0>,
                    {send_req,
                        {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false",
                             "127.0.0.1",5983,undefined,undefined,
                             "/t/doc1?new_edits=false",http,ipv4_address},
                         [{"Accept","application/json"},
                          {"Content-Length",33194202},
                          {"Content-Type",
                           "multipart/related; boundary=\"0dea87076009b928b191e0b456375c93\""},
                          {"User-Agent","CouchDB-Replicator/2.0.0"}],
                         put,
                         {#Fun<couch_replicator_api_wrap.11.59841038>,
                          {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
                           [{att,<<"att_64">>,<<"application/octet-stream">>,
                                33193656,33193656,
                                <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
                                  161,242>>,
                                1,
                                {follows,<0.8720.0>,#Ref<0.0.1.23804>},
                                identity}],
                           <<"0dea87076009b928b191e0b456375c93">>,33194202}},
                         [{response_format,binary},
                          {inactivity_timeout,30000},
                          {socket_options,[{keepalive,true},{nodelay,false}]}],
                         infinity}},
                    infinity]}}}}
      

      Expected Behaviour:

      Replication eventually succeeds.

      Appendix:

      Set up Traffic Shaping on a Mac:

      (cat /etc/pf.conf && echo "dummynet-anchor \"reptest\"" && echo "anchor \"reptest\"") | sudo pfctl -f -
      echo "dummynet in quick proto tcp from any to any port 5983 pipe 3" | sudo pfctl -a reptest -f -
      echo "dummynet out quick proto tcp from any to any port 5983 pipe 4" | sudo pfctl -a reptest -f -
      sudo dnctl pipe 3 config bw 2Mbit/s
      sudo dnctl pipe 4 config bw 2Mbit/s
      sudo pfctl -E
      

      Reset with:

      sudo pfctl -f /etc/pf.conf
      sudo dnctl flush
      
      1. replication-failure-target.log
        5 kB
        Jan Lehnardt
      2. replication-failure.log
        53 kB
        Jan Lehnardt
      3. attach_large.py
        5 kB
        Nick Vatamaniuc

        Issue Links

          Activity

          Hide
          vatamane Nick Vatamaniuc added a comment -

          From investigating it seems to be related to how long it takes for the request to complete.

          I created a "paced" python multi-part sender which PUTs an attachment over period of time. It splits it into chunks then sends those with sleep in between.

          Attached script as attach_large.py. Can run it with {[./attach_large.py --size=100000 --mintime=80}} that will put an attachment of size 100000 bytes over at least 80 seconds.

          With that code I was able to get a 500 error and I get:

          HTTP/1.1 500 Internal Server Error
          Cache-Control: must-revalidate
          Content-Length: 47
          Content-Type: application/json
          Date: Mon, 20 Feb 2017 19:27:30 GMT
          Server: CouchDB/2.0.0 (Erlang OTP/18)
          X-Couch-Request-ID: 80a6cfd301
          X-CouchDB-Body-Time: 0
          
          {"error":"unknown_error","reason":"undefined"}
          
          Show
          vatamane Nick Vatamaniuc added a comment - From investigating it seems to be related to how long it takes for the request to complete. I created a "paced" python multi-part sender which PUTs an attachment over period of time. It splits it into chunks then sends those with sleep in between. Attached script as attach_large.py. Can run it with {[./attach_large.py --size=100000 --mintime=80}} that will put an attachment of size 100000 bytes over at least 80 seconds. With that code I was able to get a 500 error and I get: HTTP/1.1 500 Internal Server Error Cache-Control: must-revalidate Content-Length: 47 Content-Type: application/json Date: Mon, 20 Feb 2017 19:27:30 GMT Server: CouchDB/2.0.0 (Erlang OTP/18) X-Couch-Request-ID: 80a6cfd301 X-CouchDB-Body-Time: 0 { "error" : "unknown_error" , "reason" : "undefined" }
          Hide
          vatamane Nick Vatamaniuc added a comment -

          Confirmed that setting fabric request_timeout to a higher value like 90000 helps with this.

          At least ./attach_large.py --size=100000 --mintime=80

          Successfully finishes while it doesn't with the default value of 60000

          Show
          vatamane Nick Vatamaniuc added a comment - Confirmed that setting fabric request_timeout to a higher value like 90000 helps with this. At least ./attach_large.py --size=100000 --mintime=80 Successfully finishes while it doesn't with the default value of 60000
          Hide
          vatamane Nick Vatamaniuc added a comment - - edited

          I think I know what is happening. MP parser is created on the coordinating node. Then we send its pid to all the other nodes via a regular update_docs fabric request. Then each node talks directly to the MP parser (via the dist protocol!) to get bytes. While that happens, nothing happens at the fabric level. So fabric's update_docs request eventually times out.

          Now fabric_doc_update does handle a {{ attachment_chunk_received }} guessing in attempt to keep the request alive after each chunk is consume, but nobody sends.

          So crossing fingers and hoping we can finally fix attachments by just having the nodes reply with that message periodically from their MP consumer callbacks.

          Show
          vatamane Nick Vatamaniuc added a comment - - edited I think I know what is happening. MP parser is created on the coordinating node. Then we send its pid to all the other nodes via a regular update_docs fabric request. Then each node talks directly to the MP parser (via the dist protocol!) to get bytes. While that happens, nothing happens at the fabric level. So fabric's update_docs request eventually times out. Now fabric_doc_update does handle a {{ attachment_chunk_received }} guessing in attempt to keep the request alive after each chunk is consume, but nobody sends. So crossing fingers and hoping we can finally fix attachments by just having the nodes reply with that message periodically from their MP consumer callbacks.
          Hide
          paul.joseph.davis Paul Joseph Davis added a comment -

          You logged that nothing sends that message? There's a call to send it in two places:

          https://github.com/apache/couchdb-fabric/blob/master/src/fabric_doc_attachments.erl#L40
          https://github.com/apache/couchdb-fabric/blob/master/src/fabric_doc_attachments.erl#L71

          This code is gnarly enough that its quite possible there's something broken with those calls obviously but given how rexi:reply/1 should throw a badmatch if the rexi_from pdict entry isn't set I'm not sure what it'd be.

          Show
          paul.joseph.davis Paul Joseph Davis added a comment - You logged that nothing sends that message? There's a call to send it in two places: https://github.com/apache/couchdb-fabric/blob/master/src/fabric_doc_attachments.erl#L40 https://github.com/apache/couchdb-fabric/blob/master/src/fabric_doc_attachments.erl#L71 This code is gnarly enough that its quite possible there's something broken with those calls obviously but given how rexi:reply/1 should throw a badmatch if the rexi_from pdict entry isn't set I'm not sure what it'd be.
          Hide
          vatamane Nick Vatamaniuc added a comment - - edited

          fabric_doc_attachments is used when PUT-ing individual attachments, I was looking at a doc PUT with attachments in mulitpart-related format

          Show
          vatamane Nick Vatamaniuc added a comment - - edited fabric_doc_attachments is used when PUT-ing individual attachments, I was looking at a doc PUT with attachments in mulitpart-related format
          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user nickva opened a pull request:

          https://github.com/apache/couchdb-fabric/pull/89

          Prevent attachment upload from timing out during update_docs fabric call

          Currently if an attachment was large enough or the connection was slow enough
          such that it took more than fabric.request_timeout = 60000 milliseconds, the
          fabric request would time out during attachment data transfer from coordinator
          node to other nodes and the whole request would fail.

          This was most evident when replicating database with large attachments.

          The fix is to periodically send `attachment_chunk_received` to coordinator to
          prevent the timeout.

          COUCHDB-3302

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cloudant/couchdb-fabric couchdb-3302

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/couchdb-fabric/pull/89.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #89


          commit 6e9074bc8778e00471d96191319ac67d6c78c05a
          Author: Nick Vatamaniuc <vatamane@apache.org>
          Date: 2017-02-21T22:46:57Z

          Prevent attachment upload from timing out during update_docs fabric call

          Currently if an attachment was large enough or the connection was slow enough
          such that it took more than fabric.request_timeout = 60000 milliseconds, the
          fabric request would time out during attachment data transfer from coordinator
          node to other nodes and the whole request would fail.

          This was most evident when replicating database with large attachments.

          The fix is to periodically send `attachment_chunk_received` to coordinator to
          prevent the timeout.

          COUCHDB-3302


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user nickva opened a pull request: https://github.com/apache/couchdb-fabric/pull/89 Prevent attachment upload from timing out during update_docs fabric call Currently if an attachment was large enough or the connection was slow enough such that it took more than fabric.request_timeout = 60000 milliseconds, the fabric request would time out during attachment data transfer from coordinator node to other nodes and the whole request would fail. This was most evident when replicating database with large attachments. The fix is to periodically send `attachment_chunk_received` to coordinator to prevent the timeout. COUCHDB-3302 You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloudant/couchdb-fabric couchdb-3302 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/couchdb-fabric/pull/89.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #89 commit 6e9074bc8778e00471d96191319ac67d6c78c05a Author: Nick Vatamaniuc <vatamane@apache.org> Date: 2017-02-21T22:46:57Z Prevent attachment upload from timing out during update_docs fabric call Currently if an attachment was large enough or the connection was slow enough such that it took more than fabric.request_timeout = 60000 milliseconds, the fabric request would time out during attachment data transfer from coordinator node to other nodes and the whole request would fail. This was most evident when replicating database with large attachments. The fix is to periodically send `attachment_chunk_received` to coordinator to prevent the timeout. COUCHDB-3302
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/master from Nick Vatamaniuc
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ]

          Prevent attachment upload from timing out during update_docs fabric call

          Currently if an attachment was large enough or the connection was slow enough
          such that it took more than fabric.request_timeout = 60000 milliseconds, the
          fabric request would time out during attachment data transfer from coordinator
          node to other nodes and the whole request would fail.

          This was most evident when replicating database with large attachments.

          The fix is to periodically send `attachment_chunk_received` to coordinator to
          prevent the timeout.

          COUCHDB-3302

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/master from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ] Prevent attachment upload from timing out during update_docs fabric call Currently if an attachment was large enough or the connection was slow enough such that it took more than fabric.request_timeout = 60000 milliseconds, the fabric request would time out during attachment data transfer from coordinator node to other nodes and the whole request would fail. This was most evident when replicating database with large attachments. The fix is to periodically send `attachment_chunk_received` to coordinator to prevent the timeout. COUCHDB-3302
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/couchdb-fabric/pull/89

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/couchdb-fabric/pull/89
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 078d46c5a18864edd2f4bc16c89f12178ce13076 in couchdb's branch refs/heads/master from Nick Vatamaniuc
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=078d46c ]

          Bump fabric : fix attachment uploading

          COUCHDB-3302

          Show
          jira-bot ASF subversion and git services added a comment - Commit 078d46c5a18864edd2f4bc16c89f12178ce13076 in couchdb's branch refs/heads/master from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb.git;h=078d46c ] Bump fabric : fix attachment uploading COUCHDB-3302
          Hide
          janl Jan Lehnardt added a comment -

          Nice, thank you!

          Show
          janl Jan Lehnardt added a comment - Nice, thank you!
          Hide
          vatamane Nick Vatamaniuc added a comment -

          Thanks for bringing it up. Your debugging logs and additional info made it easy to solve.

          Show
          vatamane Nick Vatamaniuc added a comment - Thanks for bringing it up. Your debugging logs and additional info made it easy to solve.
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/2971-count-distinct from Nick Vatamaniuc
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ]

          Prevent attachment upload from timing out during update_docs fabric call

          Currently if an attachment was large enough or the connection was slow enough
          such that it took more than fabric.request_timeout = 60000 milliseconds, the
          fabric request would time out during attachment data transfer from coordinator
          node to other nodes and the whole request would fail.

          This was most evident when replicating database with large attachments.

          The fix is to periodically send `attachment_chunk_received` to coordinator to
          prevent the timeout.

          COUCHDB-3302

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/2971-count-distinct from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ] Prevent attachment upload from timing out during update_docs fabric call Currently if an attachment was large enough or the connection was slow enough such that it took more than fabric.request_timeout = 60000 milliseconds, the fabric request would time out during attachment data transfer from coordinator node to other nodes and the whole request would fail. This was most evident when replicating database with large attachments. The fix is to periodically send `attachment_chunk_received` to coordinator to prevent the timeout. COUCHDB-3302
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/COUCHDB-3288-remove-public-db-record from Nick Vatamaniuc
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ]

          Prevent attachment upload from timing out during update_docs fabric call

          Currently if an attachment was large enough or the connection was slow enough
          such that it took more than fabric.request_timeout = 60000 milliseconds, the
          fabric request would time out during attachment data transfer from coordinator
          node to other nodes and the whole request would fail.

          This was most evident when replicating database with large attachments.

          The fix is to periodically send `attachment_chunk_received` to coordinator to
          prevent the timeout.

          COUCHDB-3302

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/ COUCHDB-3288 -remove-public-db-record from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ] Prevent attachment upload from timing out during update_docs fabric call Currently if an attachment was large enough or the connection was slow enough such that it took more than fabric.request_timeout = 60000 milliseconds, the fabric request would time out during attachment data transfer from coordinator node to other nodes and the whole request would fail. This was most evident when replicating database with large attachments. The fix is to periodically send `attachment_chunk_received` to coordinator to prevent the timeout. COUCHDB-3302
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/COUCHDB-3287-pluggable-storage-engines from Nick Vatamaniuc
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ]

          Prevent attachment upload from timing out during update_docs fabric call

          Currently if an attachment was large enough or the connection was slow enough
          such that it took more than fabric.request_timeout = 60000 milliseconds, the
          fabric request would time out during attachment data transfer from coordinator
          node to other nodes and the whole request would fail.

          This was most evident when replicating database with large attachments.

          The fix is to periodically send `attachment_chunk_received` to coordinator to
          prevent the timeout.

          COUCHDB-3302

          Show
          jira-bot ASF subversion and git services added a comment - Commit 6e9074bc8778e00471d96191319ac67d6c78c05a in couchdb-fabric's branch refs/heads/ COUCHDB-3287 -pluggable-storage-engines from Nick Vatamaniuc [ https://git-wip-us.apache.org/repos/asf?p=couchdb-fabric.git;h=6e9074b ] Prevent attachment upload from timing out during update_docs fabric call Currently if an attachment was large enough or the connection was slow enough such that it took more than fabric.request_timeout = 60000 milliseconds, the fabric request would time out during attachment data transfer from coordinator node to other nodes and the whole request would fail. This was most evident when replicating database with large attachments. The fix is to periodically send `attachment_chunk_received` to coordinator to prevent the timeout. COUCHDB-3302

            People

            • Assignee:
              Unassigned
              Reporter:
              janl Jan Lehnardt
            • Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

              Dates

              • Created:
                Updated:

                Development