Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-3302

Attachment replication over low bandwidth network connections

    Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Replication
    • Labels:
      None

      Description

      Setup:

      Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit network connection (simulated locally with traffic shaping, see way below for an example).

      git clone https://github.com/apache/couchdb.git
      cd couchdb
      ./configure --disable-docs --disable-fauxton
      make release
      cd ..
      cp -r couchdb/rel/couchdb source
      cp -r couchdb/rel/couchdb target
      
      # set up local ini: chttpd / port: 5981 / 5983
      # set up vm.args: source@hostname.local / target@hostname.local
      
      # no admins
      
      Start both CouchDB in their own terminal windows: ./bin/couchdb
      
      # create all required databases, and our `t` test database
      curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}
      
      # create 64MB attachments
      dd if=/dev/urandom of=att-64 bs=1024 count=65536
      
      # create doc on source
      curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: application/octet-stream' -d @att-64
      
      # replicate to target
      curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'
      

      With the traffic shaping in place, the replication call doesn’t return, and eventually CouchDB fails with:

      [error] 2017-02-16T17:37:30.488990Z source@hostname.local emulator -------- Error in process <0.15811.0> on node 'source@hostname.local' with exit value:
      {{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}
      
      [error] 2017-02-16T17:37:30.490610Z source@hostname.local <0.8721.0> -------- Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false" failed due to error {error,
          {'EXIT',
              {{{nocatch,{mp_parser_died,noproc}},
                [{couch_att,'-foldl/4-fun-0-',3,
                     [{file,"src/couch_att.erl"},{line,591}]},
                 {couch_att,fold_streamed_data,4,
                     [{file,"src/couch_att.erl"},{line,642}]},
                 {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
                 {couch_httpd_multipart,atts_to_mp,4,
                     [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
               {gen_server,call,
                   [<0.15778.0>,
                    {send_req,
                        {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false",
                             "127.0.0.1",5983,undefined,undefined,
                             "/t/doc1?new_edits=false",http,ipv4_address},
                         [{"Accept","application/json"},
                          {"Content-Length",33194202},
                          {"Content-Type",
                           "multipart/related; boundary=\"0dea87076009b928b191e0b456375c93\""},
                          {"User-Agent","CouchDB-Replicator/2.0.0"}],
                         put,
                         {#Fun<couch_replicator_api_wrap.11.59841038>,
                          {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
                           [{att,<<"att_64">>,<<"application/octet-stream">>,
                                33193656,33193656,
                                <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
                                  161,242>>,
                                1,
                                {follows,<0.8720.0>,#Ref<0.0.1.23804>},
                                identity}],
                           <<"0dea87076009b928b191e0b456375c93">>,33194202}},
                         [{response_format,binary},
                          {inactivity_timeout,30000},
                          {socket_options,[{keepalive,true},{nodelay,false}]}],
                         infinity}},
                    infinity]}}}}
      

      Expected Behaviour:

      Replication eventually succeeds.

      Appendix:

      Set up Traffic Shaping on a Mac:

      (cat /etc/pf.conf && echo "dummynet-anchor \"reptest\"" && echo "anchor \"reptest\"") | sudo pfctl -f -
      echo "dummynet in quick proto tcp from any to any port 5983 pipe 3" | sudo pfctl -a reptest -f -
      echo "dummynet out quick proto tcp from any to any port 5983 pipe 4" | sudo pfctl -a reptest -f -
      sudo dnctl pipe 3 config bw 2Mbit/s
      sudo dnctl pipe 4 config bw 2Mbit/s
      sudo pfctl -E
      

      Reset with:

      sudo pfctl -f /etc/pf.conf
      sudo dnctl flush
      

        Attachments

        1. attach_large.py
          5 kB
          Nick Vatamaniuc
        2. replication-failure.log
          53 kB
          Jan Lehnardt
        3. replication-failure-target.log
          5 kB
          Jan Lehnardt

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                janl Jan Lehnardt
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated: