[COUCHDB-3302] Attachment replication over low bandwidth network connections

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Component/s: Replication
• Assignee: Unassigned
• Reporter: Jan Lehnardt
• Created: 16/Feb/17 17:49
• Updated: 02/Sep/18 07:07
• Resolved: 02/Sep/18 07:07

Description

Setup:

Two CouchDB instances `source` (5981) and `target` (5983) with a 2MBit network connection (simulated locally with traffic shaping, see way below for an example).

git clone https://github.com/apache/couchdb.git
cd couchdb
./configure --disable-docs --disable-fauxton
make release
cd ..
cp -r couchdb/rel/couchdb source
cp -r couchdb/rel/couchdb target

# set up local ini: chttpd / port: 5981 / 5983
# set up vm.args: [e-mail] / [e-mail]

# no admins

Start both CouchDB in their own terminal windows: ./bin/couchdb

# create all required databases, and our `t` test database
curl -X PUT http://127.0.0.1:598{1,3}/{_users,_replicator,_global_changes,t}

# create 64MB attachments
dd if=/dev/urandom of=att-64 bs=1024 count=65536

# create doc on source
curl -X PUT http://127.0.0.1:5981/t/doc1/att_64 -H 'Content-Type: application/octet-stream' -d @att-64

# replicate to target
curl -X POST http://127.0.0.1:5981/_replicate -Hcontent-type:application/json -d '{"source":"http://127.0.0.1:5981/t","target":"http://127.0.0.1:5983/t"}'

With the traffic shaping in place, the replication call doesn’t return, and eventually CouchDB fails with:

[error] 2017-02-16T17:37:30.488990Z [e-mail] emulator -------- Error in process <0.15811.0> on node '[e-mail]' with exit value:
{{nocatch,{mp_parser_died,noproc}},[{couch_att,'-foldl/4-fun-0-',3,[{file,"src/couch_att.erl"},{line,591}]},{couch_att,fold_streamed_data,4,[{file,"src/couch_att.erl"},{line,642}]},{couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},{couch_httpd_multipart,atts_to_mp,4,[{file,"src/couch_httpd_multipart.erl"},{line,208}]}]}

[error] 2017-02-16T17:37:30.490610Z [e-mail] <0.8721.0> -------- Replicator, request PUT to "http://127.0.0.1:5983/t/doc1?new_edits=false" failed due to error {error,
    {'EXIT',
        {{{nocatch,{mp_parser_died,noproc}},
          [{couch_att,'-foldl/4-fun-0-',3,
               [{file,"src/couch_att.erl"},{line,591}]},
           {couch_att,fold_streamed_data,4,
               [{file,"src/couch_att.erl"},{line,642}]},
           {couch_att,foldl,4,[{file,"src/couch_att.erl"},{line,595}]},
           {couch_httpd_multipart,atts_to_mp,4,
               [{file,"src/couch_httpd_multipart.erl"},{line,208}]}]},
         {gen_server,call,
             [<0.15778.0>,
              {send_req,
                  {{url,"http://127.0.0.1:5983/t/doc1?new_edits=false",
                       "127.0.0.1",5983,undefined,undefined,
                       "/t/doc1?new_edits=false",http,ipv4_address},
                   [{"Accept","application/json"},
                    {"Content-Length",33194202},
                    {"Content-Type",
                     "multipart/related; boundary=\"0dea87076009b928b191e0b456375c93\""},
                    {"User-Agent","CouchDB-Replicator/2.0.0"}],
                   put,
                   {#Fun<couch_replicator_api_wrap.11.59841038>,
                    {<<"{\"_id\":\"doc1\",\"_rev\":\"1-15ae43c5b53de894b936c08db31d537c\",\"_revisions\":{\"start\":1,\"ids\":[\"15ae43c5b53de894b936c08db31d537c\"]},\"_attachments\":{\"att_64\":{\"content_type\":\"application/octet-stream\",\"revpos\":1,\"digest\":\"md5-s3AA0cYvwOzrSFTaALGh8g==\",\"length\":33193656,\"follows\":true}}}">>,
                     [{att,<<"att_64">>,<<"application/octet-stream">>,
                          33193656,33193656,
                          <<179,112,0,209,198,47,192,236,235,72,84,218,0,177,
                            161,242>>,
                          1,
                          {follows,<0.8720.0>,#Ref<0.0.1.23804>},
                          identity}],
                     <<"0dea87076009b928b191e0b456375c93">>,33194202}},
                   [{response_format,binary},
                    {inactivity_timeout,30000},
                    {socket_options,[{keepalive,true},{nodelay,false}]}],
                   infinity}},
              infinity]}}}}

Expected Behaviour:

Replication eventually succeeds.

Appendix:

Set up Traffic Shaping on a Mac:

(cat /etc/pf.conf && echo "dummynet-anchor \"reptest\"" && echo "anchor \"reptest\"") | sudo pfctl -f -
echo "dummynet in quick proto tcp from any to any port 5983 pipe 3" | sudo pfctl -a reptest -f -
echo "dummynet out quick proto tcp from any to any port 5983 pipe 4" | sudo pfctl -a reptest -f -
sudo dnctl pipe 3 config bw 2Mbit/s
sudo dnctl pipe 4 config bw 2Mbit/s
sudo pfctl -E

Reset with:

sudo pfctl -f /etc/pf.conf
sudo dnctl flush

Issue Links

• links to: GitHub Pull Request #89