Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-1946

Trying to replicate NPM grinds to a halt after 40GB

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 1.7.0, 1.6.2, 2.1.0
    • Component/s: Database Core
    • Labels:
      None

      Description

      I have been able to replicate the Node.js NPM database until 40G or so, then I get this:

      https://gist.github.com/stelcheck/7723362

      I one case I have gotten a flat-out OOM error, but I didn't take a dump of the log output at the time.

      CentOS6.4 with CouchDB 1.5 (also tried 1.3.1, but to no avail). Also tried to restart replication from scratch - twice - bot cases stalling at 40GB.

      1. couch.log
        34 kB
        Marc Trudel

        Activity

        Hide
        dch Dave Cottlehuber added a comment - - edited

        Hi Marc,

        thanks for reporting this. We'll need a bit more info to make headway on this; the error message you see is a relatively unimportant part of the Erlang VM.

        If you can start your instance up, I'm interested in a couple of specific things:

        what the last update sequence number of your registry copy is:

        export COUCH=http://localhost:5984
        curl $COUCH/registry

        should return a JSON blob like this:

        {
        "committed_update_seq": 7305,
        "compact_running": false,
        "data_size": 2040934117,
        "db_name": "registry",
        "disk_format_version": 6,
        "disk_size": 3444014974,
        "doc_count": 5208,
        "doc_del_count": 754,
        "instance_start_time": "1386014605129594",
        "purge_seq": 0,
        "update_seq": 7305
        }

        Also, if you can make the couch.log file available (privately is fine, dch@apache.org GPG key for optional signing http://people.apache.org/~dch/KEYS) we might find something more enlightening.

        Previous occurences of the error you mentioned have all been related to insufficient memory. FWIW I'm running replication < 2GB RAM atm on a gce small instance, 2 cores, and that's cpu bound only.

        You can try reducing the parallelisation of replication; reduce:

        • worker_batch_size to ease RAM pressure
        • worker_processes for disk & network IO

        Other Couch folk report reducing these right down has helped a lot initially.

        Full info is here https://couchdb.readthedocs.org/en/latest/config/replicator.html

        Current parameters can be seen via GET /_config/replicator, and you can do updates via PUT or also via futon's configuation interface. This will avoid the need to restart couchdb, which is only required if you edit the ini file by hand.

        Bear in mind that a heavy load is borne by the central registry atm, there may not be a great deal you can influence on your local node wrt to performance.

        Show
        dch Dave Cottlehuber added a comment - - edited Hi Marc, thanks for reporting this. We'll need a bit more info to make headway on this; the error message you see is a relatively unimportant part of the Erlang VM. If you can start your instance up, I'm interested in a couple of specific things: what the last update sequence number of your registry copy is: export COUCH= http://localhost:5984 curl $COUCH/registry should return a JSON blob like this: { "committed_update_seq": 7305, "compact_running": false, "data_size": 2040934117, "db_name": "registry", "disk_format_version": 6, "disk_size": 3444014974, "doc_count": 5208, "doc_del_count": 754, "instance_start_time": "1386014605129594", "purge_seq": 0, "update_seq": 7305 } Also, if you can make the couch.log file available (privately is fine, dch@apache.org GPG key for optional signing http://people.apache.org/~dch/KEYS ) we might find something more enlightening. Previous occurences of the error you mentioned have all been related to insufficient memory. FWIW I'm running replication < 2GB RAM atm on a gce small instance, 2 cores, and that's cpu bound only. You can try reducing the parallelisation of replication; reduce: worker_batch_size to ease RAM pressure worker_processes for disk & network IO Other Couch folk report reducing these right down has helped a lot initially. Full info is here https://couchdb.readthedocs.org/en/latest/config/replicator.html Current parameters can be seen via GET /_config/replicator, and you can do updates via PUT or also via futon's configuation interface. This will avoid the need to restart couchdb, which is only required if you edit the ini file by hand. Bear in mind that a heavy load is borne by the central registry atm, there may not be a great deal you can influence on your local node wrt to performance.
        Hide
        stelcheck Marc Trudel added a comment -

        curl $COUCH/registry | python -mjson.tool
        ```javascript
        {
        "committed_update_seq": 51009,
        "compact_running": false,
        "data_size": 39439860666,
        "db_name": "registry",
        "disk_format_version": 6,
        "disk_size": 47768499249,
        "doc_count": 38858,
        "doc_del_count": 3886,
        "instance_start_time": "1386033114907901",
        "purge_seq": 0,
        "update_seq": 51009
        }
        ```

        Logs are here attached. This is a log from startup to crash, so there is certainly a bit of noise in there.

        Extra note: as download goes, until I hit the 40G mark I replicate at 10Mbps. Then it goes down slowly until it hits a bit below 200Kbps

        I am downloading a .couch file copy from someone who already has a server up and running. Will give a try to this too, and will report any findings...

        Show
        stelcheck Marc Trudel added a comment - curl $COUCH/registry | python -mjson.tool ```javascript { "committed_update_seq": 51009, "compact_running": false, "data_size": 39439860666, "db_name": "registry", "disk_format_version": 6, "disk_size": 47768499249, "doc_count": 38858, "doc_del_count": 3886, "instance_start_time": "1386033114907901", "purge_seq": 0, "update_seq": 51009 } ``` Logs are here attached. This is a log from startup to crash, so there is certainly a bit of noise in there. Extra note: as download goes, until I hit the 40G mark I replicate at 10Mbps. Then it goes down slowly until it hits a bit below 200Kbps I am downloading a .couch file copy from someone who already has a server up and running. Will give a try to this too, and will report any findings...
        Hide
        MiLk Emilien Kenler added a comment - - edited

        Hi,

        I tried to do a npm replica too.

        I set up a new VM with 1GB RAM on Debian Wheezy with CouchDB 1.2.0 from repository and Erlang R15B01 (erts-5.9.1).
        I create the replica by using the following command:

        curl -X POST http://127.0.0.1:5984/_replicate -d '

        {"source":"http://isaacs.iriscouch.com/registry/", "target":"registry", "continuous":true, "create_target":true}

        ' -H "Content-Type: application/json"

        In about 12h, I obtained a working replica (127GB).
        Then I saw the Marc's problem, and noticed that npm is on CouchDB 1.5.0.
        I decided to run a new replica on a new VM with the same configuration exept CouchDB in 1.5.0.

        I followed the instructions in the documentation and on the wiki, and I installed the libmozjs185-cloudant

        {,-dev}

        packages. http://wiki.apache.org/couchdb/Installing_on_Debian

        I start the replication from my working CouchDB 1.2.0 server to the new CouchDB 1.5.0 server.
        In few hours (about 36GB replicated), the replication stopped and it crashed (Out of memory in syslog).

        I upgraded my VM to 8GB of memory, but nothing changes.

        The error logs are here: https://gist.github.com/MiLk/a1abd11aedb24c20c9d5

        Show
        MiLk Emilien Kenler added a comment - - edited Hi, I tried to do a npm replica too. I set up a new VM with 1GB RAM on Debian Wheezy with CouchDB 1.2.0 from repository and Erlang R15B01 (erts-5.9.1). I create the replica by using the following command: curl -X POST http://127.0.0.1:5984/_replicate -d ' {"source":"http://isaacs.iriscouch.com/registry/", "target":"registry", "continuous":true, "create_target":true} ' -H "Content-Type: application/json" In about 12h, I obtained a working replica (127GB). Then I saw the Marc's problem, and noticed that npm is on CouchDB 1.5.0. I decided to run a new replica on a new VM with the same configuration exept CouchDB in 1.5.0. I followed the instructions in the documentation and on the wiki, and I installed the libmozjs185-cloudant {,-dev} packages. http://wiki.apache.org/couchdb/Installing_on_Debian I start the replication from my working CouchDB 1.2.0 server to the new CouchDB 1.5.0 server. In few hours (about 36GB replicated), the replication stopped and it crashed (Out of memory in syslog). I upgraded my VM to 8GB of memory, but nothing changes. The error logs are here: https://gist.github.com/MiLk/a1abd11aedb24c20c9d5
        Hide
        dch Dave Cottlehuber added a comment -

        hey folks,

        Something does happen around 30-40GB, I have replicated the situation here.

        In both your issues, the final error is termination of couchjs process (used for map/reduce and any
        javascripty stuff) by OOM – you can see things like this, however that's not the root cause.

        [Tue, 03 Dec 2013 01:47:46 GMT] [error] [<0.368.0>] OS Process died with status: 137
        [Tue, 03 Dec 2013 01:47:47 GMT] [error] [<0.368.0>] ** Generic server <0.368.0> terminating 
        ** Last message in was {#Port<0.2771>,{exit_status,137}}
        ** When Server state == {os_proc,"/usr/bin/couchjs /usr/share/couchdb/server/main.js",
                                         #Port<0.2771>,
                                         #Fun<couch_os_process.2.132569728>,
                                         #Fun<couch_os_process.3.35601548>,5000}
        

        This is the issue you're hitting, and you need to at least ensure ulimits etc
        are set appropriately for the couchdb user:

        http://wiki.apache.org/couchdb/Performance

        You can optionally force OOMkiller off for the beam.smp process directly;

        # assuming you run only one erlang VM at a time
        echo '-1000' > /proc/`pgrep beam.smp`/oom_score_adj
        

        It can be helpful to increase os process timeout, although that's not
        what you're hitting here from what I see.

         curl -XPUT https://admin:passwd@localhost:5984/_config/couchdb/os_process_timeout -d '"60000"'
        

        However, there is some other issue here, before the OOM killer goes awol, CouchDB
        starts consuming a lot of memory. My instance cruises along < 300MB RES RAM for
        most of the replication, and then starts shooting up very rapidly past 3GB and then its
        all over red rover. I'm working on tracking down exactly what this is but it's somewhat
        tricky given the amount of concurrent stuff happening.

          Node: npm@z1 (Connected) (R15B01/5.9.1) unix (linux 3.2.0) CPU:4 SMP +A:16 +K
        Time: local time 15:04:41, up for 000:00:33:51, 0ms latency,
        Processes: total 504 (RQ 42) at 53424637 RpI using 63038.5k (63084.9k allocated)
        Memory: Sys 2490.3m, Atom 264.9k/266.3k, Bin 2480.2m, Code 6292.2k, Ets 885.3k
        
        Interval 1000ms, Sorting on "HTot" (Descending), Retrieved in 15ms
                 Pid Registered Name      Reductions   MQueue HSize  SSize  HTot
          <0.4203.0> -                    459031       0      514229 9      832040
          <0.5277.0> -                    31099        0      121393 19     439204
             <0.6.0> error_logger         17874687     0      46368  8      364179
          <0.5840.0> -                    367940       1      28657  189    225075
          <0.6295.0> -
        
        
        Node: npm@z1 (Connected) (R15B01/5.9.1) unix (linux 3.2.0) CPU:4 SMP +A:16 +K
        Time: local time 15:04:22, up for 000:00:33:32, 0ms latency,
        Processes: total 504 (RQ 46) at 53654512 RpI using 62889.0k (62899.1k allocated)
        Memory: Sys 2075.6m, Atom 264.9k/266.3k, Bin 2065.5m, Code 6292.2k, Ets 880.0k
        
        Interval 1000ms, Sorting on "SSize" (Descending), Retrieved in 17ms
                 Pid Registered Name      Reductions   MQueue HSize  SSize  HTot
          <0.6295.0> -                    351722       0      28657  204    225075
          <0.5840.0> -                    340874       0      28657  158    225075
          <0.5063.0> -                    302674       1      17711  118    139104
          <0.4434.0> -                    69314        0      10946  86     85971
          <0.5062.0> -
          
          
        Node: npm@z1 (Connected) (R15B01/5.9.1) unix (linux 3.2.0) CPU:4 SMP +A:16 +K
        Time: local time 15:06:02, up for 000:00:35:12, 0ms latency,
        Processes: total 496 (RQ 42) at 54048301 RpI using 58790.1k (58810.5k allocated)
        Memory: Sys 2904.0m, Atom 264.9k/266.3k, Bin 2893.9m, Code 6292.2k, Ets 908.1k
        
        Interval 1000ms, Sorting on "HSize" (Descending), Retrieved in 18ms
                 Pid Registered Name      Reductions   MQueue HSize  SSize  HTot
          <0.4203.0> -                    459031       0      514229 9      832040
          <0.5002.0> -                    31564        0      196418 19     196418
          <0.5663.0> -                    36248        0      196418 19     196418
            <0.12.0> rex                  1330480      0      121393 9      121770
          <0.3919.0> couch_stats_a
          

        The last 15 minutes before the crash, the message queue for one process appears stuck
        at 10: `MQueue 10 <0.4203.0>` which could be normal or not, but I do think its weird.

        atm npm itself is not letting me replicate so I can't debug further until its available again.

        Show
        dch Dave Cottlehuber added a comment - hey folks, Something does happen around 30-40GB, I have replicated the situation here. In both your issues, the final error is termination of couchjs process (used for map/reduce and any javascripty stuff) by OOM – you can see things like this, however that's not the root cause. [Tue, 03 Dec 2013 01:47:46 GMT] [error] [<0.368.0>] OS Process died with status: 137 [Tue, 03 Dec 2013 01:47:47 GMT] [error] [<0.368.0>] ** Generic server <0.368.0> terminating ** Last message in was {#Port<0.2771>,{exit_status,137}} ** When Server state == {os_proc, "/usr/bin/couchjs /usr/share/couchdb/server/main.js" , #Port<0.2771>, #Fun<couch_os_process.2.132569728>, #Fun<couch_os_process.3.35601548>,5000} This is the issue you're hitting, and you need to at least ensure ulimits etc are set appropriately for the couchdb user: http://wiki.apache.org/couchdb/Performance You can optionally force OOMkiller off for the beam.smp process directly; # assuming you run only one erlang VM at a time echo '-1000' > /proc/`pgrep beam.smp`/oom_score_adj It can be helpful to increase os process timeout, although that's not what you're hitting here from what I see. curl -XPUT https: //admin:passwd@localhost:5984/_config/couchdb/os_process_timeout -d ' "60000" ' However, there is some other issue here, before the OOM killer goes awol, CouchDB starts consuming a lot of memory. My instance cruises along < 300MB RES RAM for most of the replication, and then starts shooting up very rapidly past 3GB and then its all over red rover. I'm working on tracking down exactly what this is but it's somewhat tricky given the amount of concurrent stuff happening. Node: npm@z1 (Connected) (R15B01/5.9.1) unix (linux 3.2.0) CPU:4 SMP +A:16 +K Time: local time 15:04:41, up for 000:00:33:51, 0ms latency, Processes: total 504 (RQ 42) at 53424637 RpI using 63038.5k (63084.9k allocated) Memory: Sys 2490.3m, Atom 264.9k/266.3k, Bin 2480.2m, Code 6292.2k, Ets 885.3k Interval 1000ms, Sorting on "HTot" (Descending), Retrieved in 15ms Pid Registered Name Reductions MQueue HSize SSize HTot <0.4203.0> - 459031 0 514229 9 832040 <0.5277.0> - 31099 0 121393 19 439204 <0.6.0> error_logger 17874687 0 46368 8 364179 <0.5840.0> - 367940 1 28657 189 225075 <0.6295.0> - Node: npm@z1 (Connected) (R15B01/5.9.1) unix (linux 3.2.0) CPU:4 SMP +A:16 +K Time: local time 15:04:22, up for 000:00:33:32, 0ms latency, Processes: total 504 (RQ 46) at 53654512 RpI using 62889.0k (62899.1k allocated) Memory: Sys 2075.6m, Atom 264.9k/266.3k, Bin 2065.5m, Code 6292.2k, Ets 880.0k Interval 1000ms, Sorting on "SSize" (Descending), Retrieved in 17ms Pid Registered Name Reductions MQueue HSize SSize HTot <0.6295.0> - 351722 0 28657 204 225075 <0.5840.0> - 340874 0 28657 158 225075 <0.5063.0> - 302674 1 17711 118 139104 <0.4434.0> - 69314 0 10946 86 85971 <0.5062.0> - Node: npm@z1 (Connected) (R15B01/5.9.1) unix (linux 3.2.0) CPU:4 SMP +A:16 +K Time: local time 15:06:02, up for 000:00:35:12, 0ms latency, Processes: total 496 (RQ 42) at 54048301 RpI using 58790.1k (58810.5k allocated) Memory: Sys 2904.0m, Atom 264.9k/266.3k, Bin 2893.9m, Code 6292.2k, Ets 908.1k Interval 1000ms, Sorting on "HSize" (Descending), Retrieved in 18ms Pid Registered Name Reductions MQueue HSize SSize HTot <0.4203.0> - 459031 0 514229 9 832040 <0.5002.0> - 31564 0 196418 19 196418 <0.5663.0> - 36248 0 196418 19 196418 <0.12.0> rex 1330480 0 121393 9 121770 <0.3919.0> couch_stats_a The last 15 minutes before the crash, the message queue for one process appears stuck at 10: `MQueue 10 <0.4203.0>` which could be normal or not, but I do think its weird. atm npm itself is not letting me replicate so I can't debug further until its available again.
        Hide
        stelcheck Marc Trudel added a comment -

        I have a full replicate running now, thanks to Emilien. Please PM me if you
        want access to it to test replication.


        <http://www.wizcorp.jp/>Marc Trudel-Belisle
        Chief Technology Officer | Wizcorp Inc. <http://www.wizcorp.jp/>
        ------------------------------
        TECH . GAMING . OPEN-SOURCE WIZARDS+ 81
        3-4550-1448|Website<http://www.wizcorp.jp/>

        Show
        stelcheck Marc Trudel added a comment - I have a full replicate running now, thanks to Emilien. Please PM me if you want access to it to test replication. – < http://www.wizcorp.jp/ >Marc Trudel-Belisle Chief Technology Officer | Wizcorp Inc. < http://www.wizcorp.jp/ > ------------------------------ TECH . GAMING . OPEN-SOURCE WIZARDS+ 81 3-4550-1448|Website< http://www.wizcorp.jp/ >
        Hide
        kocolosk Adam Kocoloski added a comment -

        Hmm, this is certainly strange. Does restarting the replication after the crash cause it to crash again?

        If you happen to have an Erlang shell on the node when it's starts leaking memory I'd be curious to see the output of something like

        lists:reverse(lists:sort([

        {process_info(P, [memory,initial_call,dictionary]), P}

        || P <- processes()])).

        Show
        kocolosk Adam Kocoloski added a comment - Hmm, this is certainly strange. Does restarting the replication after the crash cause it to crash again? If you happen to have an Erlang shell on the node when it's starts leaking memory I'd be curious to see the output of something like lists:reverse(lists:sort([ {process_info(P, [memory,initial_call,dictionary]), P} || P <- processes()])).
        Hide
        dch Dave Cottlehuber added a comment -

        Hey Adam, thanks. Yes it does, repeatedly. Takes around mmm 10-30 minutes to bloat. Any other pro debugging tips appreciated! I've not got time to work on this much today, more news on Friday.

        Show
        dch Dave Cottlehuber added a comment - Hey Adam, thanks. Yes it does, repeatedly. Takes around mmm 10-30 minutes to bloat. Any other pro debugging tips appreciated! I've not got time to work on this much today, more news on Friday.
        Hide
        kocolosk Adam Kocoloski added a comment -

        Dave Cottlehuber was kind enough to allow me onto his VM this afternoon to poke around. Thanks Dave! Here's what I found:

        1. The memory utilization is all taken up in refc binaries, not processes.
        2. The binaries are mostly attached to couch_stream processes.
        3. Adding a hibernate after each write by the stream process goes a long way towards stabilizing memory usage.

        For posterity here's how you sum up the size of all binaries attached to a process P:

        BinMem = fun(P) -> case  process_info(P, binary) of {binary, Bins} ->  lists:sum([Size || {_, Size, _} <- Bins]); _ -> 0 end end.
        

        and here's the diff against 1.5.0 to cause couch_stream to hibernate after each write:

        diff --git a/src/couchdb/couch_stream.erl b/src/couchdb/couch_stream.erl
        index 959feef..4067ff7 100644
        --- a/src/couchdb/couch_stream.erl
        +++ b/src/couchdb/couch_stream.erl
        @@ -255,7 +255,7 @@ handle_call({write, Bin}, _From, Stream) ->
                                 buffer_len=0,
                                 md5=Md5_2,
                                 identity_md5=IdenMd5_2,
        -                        identity_len=IdenLen + BinSize}};
        +                        identity_len=IdenLen + BinSize}, hibernate};
             true ->
                 {reply, ok, Stream#stream{
                                 buffer_list=[Bin|Buffer],
        

        Adding that patch will cause an increase in CPU consumption when writing attachments. There may well be more subtle changes (e.g. playing with the fullsweep_after option when starting the streamer) that could achieve stability with fewer CPU cycles.

        I should also note that while memory usage is far more stable it is still sitting at 2.2 GB RES right now and seems to be gradually climbing over time, so don't go replicating to a t1.micro instance just yet. I think we do have a relatively good understanding of what's going on at this point, though.

        Show
        kocolosk Adam Kocoloski added a comment - Dave Cottlehuber was kind enough to allow me onto his VM this afternoon to poke around. Thanks Dave! Here's what I found: The memory utilization is all taken up in refc binaries, not processes. The binaries are mostly attached to couch_stream processes. Adding a hibernate after each write by the stream process goes a long way towards stabilizing memory usage. For posterity here's how you sum up the size of all binaries attached to a process P: BinMem = fun(P) -> case process_info(P, binary) of {binary, Bins} -> lists:sum([Size || {_, Size, _} <- Bins]); _ -> 0 end end. and here's the diff against 1.5.0 to cause couch_stream to hibernate after each write: diff --git a/src/couchdb/couch_stream.erl b/src/couchdb/couch_stream.erl index 959feef..4067ff7 100644 --- a/src/couchdb/couch_stream.erl +++ b/src/couchdb/couch_stream.erl @@ -255,7 +255,7 @@ handle_call({write, Bin}, _From, Stream) -> buffer_len=0, md5=Md5_2, identity_md5=IdenMd5_2, - identity_len=IdenLen + BinSize}}; + identity_len=IdenLen + BinSize}, hibernate}; true -> {reply, ok, Stream#stream{ buffer_list=[Bin|Buffer], Adding that patch will cause an increase in CPU consumption when writing attachments. There may well be more subtle changes (e.g. playing with the fullsweep_after option when starting the streamer) that could achieve stability with fewer CPU cycles. I should also note that while memory usage is far more stable it is still sitting at 2.2 GB RES right now and seems to be gradually climbing over time, so don't go replicating to a t1.micro instance just yet. I think we do have a relatively good understanding of what's going on at this point, though.
        Hide
        kocolosk Adam Kocoloski added a comment -

        I played with setting the {fullsweep_after, 0} option on couch_file and couch_stream. It didn't have much of an effect on couch_stream (not surprising since I was already hibernating the server after each write, but it reduced the memory consumption of the couch_file down to ~nothing.

        The reason for the couch_stream refc binary memory consumption continues to elude me. I think the next step may to head up the stack towards the replicator processes. Still, progress.

        Show
        kocolosk Adam Kocoloski added a comment - I played with setting the { fullsweep_after, 0 } option on couch_file and couch_stream . It didn't have much of an effect on couch_stream (not surprising since I was already hibernating the server after each write, but it reduced the memory consumption of the couch_file down to ~nothing. The reason for the couch_stream refc binary memory consumption continues to elude me. I think the next step may to head up the stack towards the replicator processes. Still, progress.
        Hide
        thor.lange Thor Anker Lange added a comment - - edited

        I am seeing the exact same behaviour when reaching 30-40 GB running CouchDB (1.5.0) on Windows (yes, unfortunally I do not have a choice of server OS). The source start sequence is around 701252.

        Marc, how did you manage to get a complete replica of the npm registry?

        Show
        thor.lange Thor Anker Lange added a comment - - edited I am seeing the exact same behaviour when reaching 30-40 GB running CouchDB (1.5.0) on Windows (yes, unfortunally I do not have a choice of server OS). The source start sequence is around 701252. Marc, how did you manage to get a complete replica of the npm registry?
        Hide
        stelcheck Marc Trudel added a comment -

        I downloaded the .couch file from someone

        I'd really suggest to people running NPM to make nightlies of this file,
        and make it available for download. Regardless of this bug, it would be
        much faster than starting with replication (and pretty similar to standard
        procedures for setting replicates in other DBMS, like MySQL)


        <http://www.wizcorp.jp/>Marc Trudel-Belisle
        Chief Technology Officer | Wizcorp Inc. <http://www.wizcorp.jp/>
        ------------------------------
        TECH . GAMING . OPEN-SOURCE WIZARDS+ 81
        3-4550-1448|Website<http://www.wizcorp.jp/>

        Show
        stelcheck Marc Trudel added a comment - I downloaded the .couch file from someone I'd really suggest to people running NPM to make nightlies of this file, and make it available for download. Regardless of this bug, it would be much faster than starting with replication (and pretty similar to standard procedures for setting replicates in other DBMS, like MySQL) – < http://www.wizcorp.jp/ >Marc Trudel-Belisle Chief Technology Officer | Wizcorp Inc. < http://www.wizcorp.jp/ > ------------------------------ TECH . GAMING . OPEN-SOURCE WIZARDS+ 81 3-4550-1448|Website< http://www.wizcorp.jp/ >
        Hide
        kxepal Alexander Shorin added a comment -

        Hey, Marc Trudel, Thor Anker Lange, Emilien Kenler

        Are you using default replicator settings? Just to be sure that we are all on the same configuration since following Dave Cottlehuber suggestion from first post helps a lot with reducing system resources usage during replication.

        Show
        kxepal Alexander Shorin added a comment - Hey, Marc Trudel , Thor Anker Lange , Emilien Kenler Are you using default replicator settings? Just to be sure that we are all on the same configuration since following Dave Cottlehuber suggestion from first post helps a lot with reducing system resources usage during replication.
        Hide
        stelcheck Marc Trudel added a comment -

        I tried all sorts of settings, but always ended up with the same outcome.
        Just took more or less time.


        <http://www.wizcorp.jp/>Marc Trudel-Belisle
        Chief Technology Officer | Wizcorp Inc. <http://www.wizcorp.jp/>
        ------------------------------
        TECH . GAMING . OPEN-SOURCE WIZARDS+ 81
        3-4550-1448|Website<http://www.wizcorp.jp/>

        Show
        stelcheck Marc Trudel added a comment - I tried all sorts of settings, but always ended up with the same outcome. Just took more or less time. – < http://www.wizcorp.jp/ >Marc Trudel-Belisle Chief Technology Officer | Wizcorp Inc. < http://www.wizcorp.jp/ > ------------------------------ TECH . GAMING . OPEN-SOURCE WIZARDS+ 81 3-4550-1448|Website< http://www.wizcorp.jp/ >
        Hide
        dch Dave Cottlehuber added a comment - - edited

        Marc Trudel agreed
        Thor Anker Lange

        There's something with replicating this specific doc that seems to trigger issues. Here's what I used to identify it (call source db and use since= <checkpoint -1)

        http://isaacs.iriscouch.com/registry/_changes&#63;limit\=2\&since\=701251

        here's some things you can try:

        1. option 1
        • delete all existing replications
        • compact your DB if there's a big difference between data size and on-disk size. jq is awesome for this.

        curl -s http://localhost:5984/registry | jq ' (.disk_size| tonumber) - (.data_size |tonumber)'

        http://stedolan.github.io/jq/

        This is a good spot to copy the registry.couch file if you have space, in case you need to revert back to it.

        • replicate the single failing document by POSTing this to _replicator. This could take a while.
        {
           "source": "http://isaacs.iriscouch.com/registry",
           "target": "registry",
           "doc_ids": [
               "as-stream"
           ],
          "user_ctx": {"roles":["_admin"]}
        }
        
        • this is simply replicating the single stuck document. If you do this, I would love an ngrep or tcpdump of the traffic to see what happens on the wire during these stuck transfers
        • once this is completed, you can then run the normal replication again.

        In 1.3 & higher, you can turn debugging on for just the replicator like so:

        curl -vXPUT http://admin:passwd@localhost:5984/_config/log/level_by_module/couch_replicator --data-binary '"debug"'
        
        1. option 2

        Install an older release of CouchDB and see if it doesn't get stuck here:

        https://archive.apache.org/dist/couchdb/binary/win/1.2.2/

        If you can please try the R15B03-1 release first, report back, and then the R14B04 one. It's not yet clear to me if the issue we are seeing is also related to garbage collection differences in Erlang/OTP between releases, or solely within CouchDB.

        1. option 3

        Sometime later (hopefully today), I should have a bitttorrent accessible version of npm. I need to update & compact first, this is pretty much IO limited .

        Show
        dch Dave Cottlehuber added a comment - - edited Marc Trudel agreed Thor Anker Lange There's something with replicating this specific doc that seems to trigger issues. Here's what I used to identify it (call source db and use since= <checkpoint -1) http://isaacs.iriscouch.com/registry/_changes&#63;limit\=2\&since\=701251 here's some things you can try: option 1 delete all existing replications compact your DB if there's a big difference between data size and on-disk size. jq is awesome for this. curl -s http://localhost:5984/registry | jq ' (.disk_size| tonumber) - (.data_size |tonumber)' http://stedolan.github.io/jq/ This is a good spot to copy the registry.couch file if you have space, in case you need to revert back to it. replicate the single failing document by POSTing this to _replicator. This could take a while . { "source" : "http: //isaacs.iriscouch.com/registry" , "target" : "registry" , "doc_ids" : [ "as-stream" ], "user_ctx" : { "roles" :[ "_admin" ]} } this is simply replicating the single stuck document. If you do this, I would love an ngrep or tcpdump of the traffic to see what happens on the wire during these stuck transfers once this is completed, you can then run the normal replication again. In 1.3 & higher, you can turn debugging on for just the replicator like so: curl -vXPUT http: //admin:passwd@localhost:5984/_config/log/level_by_module/couch_replicator --data-binary ' "debug" ' option 2 Install an older release of CouchDB and see if it doesn't get stuck here: https://archive.apache.org/dist/couchdb/binary/win/1.2.2/ If you can please try the R15B03-1 release first, report back, and then the R14B04 one. It's not yet clear to me if the issue we are seeing is also related to garbage collection differences in Erlang/OTP between releases, or solely within CouchDB. option 3 Sometime later (hopefully today), I should have a bitttorrent accessible version of npm. I need to update & compact first, this is pretty much IO limited .
        Hide
        thor.lange Thor Anker Lange added a comment - - edited

        I have been using the default replicator settings before I started trying to adjust the batch size - which did not help. Here is the last setup I tried:

        {
         "_id": "npmjs_repl",
         "source":"http://isaacs.iriscouch.com/registry/",
         "target":"registry",
         "continuous":false,
         "worker_batch_size":50,
         "user_ctx": {"name":"xpaadmin", "roles":["_admin"]}
        }
        

        I hope to try out some of the suggestions from the previous comment later today. I was also thinking of trying a downgrade of CouchDB - I have tried both R16B01 and R14B04 with the 1.5.0 version.

        Show
        thor.lange Thor Anker Lange added a comment - - edited I have been using the default replicator settings before I started trying to adjust the batch size - which did not help. Here is the last setup I tried: { "_id" : "npmjs_repl" , "source" : "http: //isaacs.iriscouch.com/registry/" , "target" : "registry" , "continuous" : false , "worker_batch_size" :50, "user_ctx" : { "name" : "xpaadmin" , "roles" :[ "_admin" ]} } I hope to try out some of the suggestions from the previous comment later today. I was also thinking of trying a downgrade of CouchDB - I have tried both R16B01 and R14B04 with the 1.5.0 version.
        Hide
        thor.lange Thor Anker Lange added a comment -

        I am doing an replication with CouchDB 1.2.2-R15B03 and this seems to have gotten past the document causing problems - currently at seq 772595.

        Show
        thor.lange Thor Anker Lange added a comment - I am doing an replication with CouchDB 1.2.2-R15B03 and this seems to have gotten past the document causing problems - currently at seq 772595.
        Hide
        dch Dave Cottlehuber added a comment -

        I'm pretty much finished a full replication of the registry with 1.5.0, multiple pipelines etc, filling my adsl link as much as possible, and I've not had any issues. Maximum RAM went up to ~ 12GiB (I have plenty to spare) but no hitches, need to copy individual docs across, nor restart replication, despite many transient network issues & apparent npm disconnects.

        What's people's thoughts on this?

        • is this a bug or simply we propose some tuning enhancements?
        • we could clearly reduce memory consumption though as Adam Kocoloski pointed out this will likely come with a cpu hit
        Show
        dch Dave Cottlehuber added a comment - I'm pretty much finished a full replication of the registry with 1.5.0, multiple pipelines etc, filling my adsl link as much as possible, and I've not had any issues. Maximum RAM went up to ~ 12GiB (I have plenty to spare) but no hitches, need to copy individual docs across, nor restart replication, despite many transient network issues & apparent npm disconnects. What's people's thoughts on this? is this a bug or simply we propose some tuning enhancements? we could clearly reduce memory consumption though as Adam Kocoloski pointed out this will likely come with a cpu hit
        Hide
        MiLk Emilien Kenler added a comment -

        Hi,

        As I said, I was able to replicate the npm registry with only 1GB of memory on CouchDB 1.2.0.
        Even if the replication works on CouchDB 1.5.0, it should work with a similar amount of memory.
        I don't think that it's an Erlang related issue, because I use the same version of Erlang for my tests.
        I use the default configuration of CouchDB, so I think it's strange to have to change something to make the replication work on 1.5.0 if it works without any change on 1.2.0.

        I can provide access to my two test VM, and I can provide new VM if needed.

        Show
        MiLk Emilien Kenler added a comment - Hi, As I said, I was able to replicate the npm registry with only 1GB of memory on CouchDB 1.2.0. Even if the replication works on CouchDB 1.5.0, it should work with a similar amount of memory. I don't think that it's an Erlang related issue, because I use the same version of Erlang for my tests. I use the default configuration of CouchDB, so I think it's strange to have to change something to make the replication work on 1.5.0 if it works without any change on 1.2.0. I can provide access to my two test VM, and I can provide new VM if needed.
        Hide
        nicknorth Nick North added a comment -

        I agree with Emilien: even if it doesn't actually break on 1.5, there's something wrong if it needs 12GB of memory now when it was fine with 1GB before, so it should be regarded as a bug. The NPM registry is one of the most prominent uses of CouchDb and, if it doesn't work well there, it dents public confidence.

        Show
        nicknorth Nick North added a comment - I agree with Emilien: even if it doesn't actually break on 1.5, there's something wrong if it needs 12GB of memory now when it was fine with 1GB before, so it should be regarded as a bug. The NPM registry is one of the most prominent uses of CouchDb and, if it doesn't work well there, it dents public confidence.
        Hide
        indexzero Charlie Robbins added a comment -

        Was able to reproduce this in a similar, but distinctly different use-case. The replication of the registry for one of our unused private replicas was not restarted for a couple of weeks. This led to the a disk_size difference between the replica and the public npm registry of >40GB.

        After restarting the replication we are now seeing this same behavior. Since these registries were up to date when this issue was opened they were already past the document that Dave mentioned. Looks to be a particular class of document(s) that causes this, not just a single document.

        Dave: is there a way to figure out which document(s) could be causing this?

        Show
        indexzero Charlie Robbins added a comment - Was able to reproduce this in a similar, but distinctly different use-case. The replication of the registry for one of our unused private replicas was not restarted for a couple of weeks. This led to the a disk_size difference between the replica and the public npm registry of >40GB. After restarting the replication we are now seeing this same behavior. Since these registries were up to date when this issue was opened they were already past the document that Dave mentioned. Looks to be a particular class of document(s) that causes this, not just a single document. Dave: is there a way to figure out which document(s) could be causing this?
        Hide
        janl Jan Lehnardt added a comment -

        Charlie Robbins Are you replicating form a single CouchDB or through a load balancer with multiple couches behind it?

        Show
        janl Jan Lehnardt added a comment - Charlie Robbins Are you replicating form a single CouchDB or through a load balancer with multiple couches behind it?
        Hide
        indexzero Charlie Robbins added a comment -

        Jan Lehnardt Hitting the individual master CouchDBs directly. Not going through the load balancer.

        Show
        indexzero Charlie Robbins added a comment - Jan Lehnardt Hitting the individual master CouchDBs directly. Not going through the load balancer.
        Hide
        indexzero Charlie Robbins added a comment -

        Jan Lehnardt Some more data about this:

        • Replication got stuck at update_seq 861259 from the public npm registry
        • Replacing the registry with a new .couch file via rsync solved the replication problem
        • We had another local replication of the registry running, and against the new .couch file local replication ran just fine past 861259

        Any material differences / optimizations for local vs. remote replications?

        Show
        indexzero Charlie Robbins added a comment - Jan Lehnardt Some more data about this: Replication got stuck at update_seq 861259 from the public npm registry Replacing the registry with a new .couch file via rsync solved the replication problem We had another local replication of the registry running, and against the new .couch file local replication ran just fine past 861259 Any material differences / optimizations for local vs. remote replications?
        Hide
        kocolosk Adam Kocoloski added a comment -

        Hi Charlie Robbins, there are absolutely material differences for local vs. remote replications – that's one axis I've been meaning to explore as soon as I have a chance to re-engage on this ticket. For example, I've not seen this happen on a replication to Cloudant (which has no real notion of a "local" replication).

        The other thing I'd love to see happen is a narrowing down of the list of affected versions; 1.2.0..1.5.0 is an awfully large changeset to bisect.

        Show
        kocolosk Adam Kocoloski added a comment - Hi Charlie Robbins , there are absolutely material differences for local vs. remote replications – that's one axis I've been meaning to explore as soon as I have a chance to re-engage on this ticket. For example, I've not seen this happen on a replication to Cloudant (which has no real notion of a "local" replication). The other thing I'd love to see happen is a narrowing down of the list of affected versions; 1.2.0..1.5.0 is an awfully large changeset to bisect.
        Hide
        mastercactapus Nathan Caza added a comment -

        So, I've been experiencing this, I've had a script replicating 1-by-1 (no bulk api). I haven't had any issues, EXCEPT as replication goes on, small documents wiz by and now towards the end, all that is left is a large number of bigger documents.

        I skipped large documents the first pass, but I presume had I not, I would have ran out of sockets/workers as they slowly filled up with the large transfers/documents.

        Also to note that one of these, as an example, is about ~40MB (a unicode document/package I believe) , when simply gzipped becomes ~1.3MB.

        Are there plans to support gzip compression?

        Show
        mastercactapus Nathan Caza added a comment - So, I've been experiencing this, I've had a script replicating 1-by-1 (no bulk api). I haven't had any issues, EXCEPT as replication goes on, small documents wiz by and now towards the end, all that is left is a large number of bigger documents. I skipped large documents the first pass, but I presume had I not, I would have ran out of sockets/workers as they slowly filled up with the large transfers/documents. Also to note that one of these, as an example, is about ~40MB (a unicode document/package I believe) , when simply gzipped becomes ~1.3MB. Are there plans to support gzip compression?
        Hide
        mastercactapus Nathan Caza added a comment -

        I should probably mention, this is on 1.5.0

        Upon further investigation, I found there are far larger documents, for instance:
        cordova@258-f88ea93535011521e52219071ebba13b

        with attachments the request is 2336256011 bytes, or 2.18GiB.
        Depending on how these get pushed in, I imagine a few packages like that (the next largest is ~1.6GiB, 1.3GiB, etc..) being replicated in parallel would explain the large memory usage, and may be the cause of this issue. And could also explain why 12G of ram seemed to handle it.

        Show
        mastercactapus Nathan Caza added a comment - I should probably mention, this is on 1.5.0 Upon further investigation, I found there are far larger documents, for instance: cordova@258-f88ea93535011521e52219071ebba13b with attachments the request is 2336256011 bytes, or 2.18GiB. Depending on how these get pushed in, I imagine a few packages like that (the next largest is ~1.6GiB, 1.3GiB, etc..) being replicated in parallel would explain the large memory usage, and may be the cause of this issue. And could also explain why 12G of ram seemed to handle it.
        Hide
        rnewson Robert Newson added a comment -

        We should enhance the replicator to ship attachments with compression if it doesn't already do so.

        Show
        rnewson Robert Newson added a comment - We should enhance the replicator to ship attachments with compression if it doesn't already do so.
        Hide
        kocolosk Adam Kocoloski added a comment -

        I guess that's COUCH-448. Bit of an oversight related to the complexity of doing multipart uploads.

        Show
        kocolosk Adam Kocoloski added a comment - I guess that's COUCH-448. Bit of an oversight related to the complexity of doing multipart uploads.
        Hide
        adamjcooper Adam Cooper added a comment -

        After hours of frustration trying to setup an npm registry with CouchDB 1.5.0, I'm glad to have found this thread to know it isn't just me.

        The only way I was able to replicate https://fullfatdb.npmjs.com/registry was by installing CouchDB 1.2.0. Is this the current recommendation until this bug is resolved?

        Thanks!

        Show
        adamjcooper Adam Cooper added a comment - After hours of frustration trying to setup an npm registry with CouchDB 1.5.0, I'm glad to have found this thread to know it isn't just me. The only way I was able to replicate https://fullfatdb.npmjs.com/registry was by installing CouchDB 1.2.0. Is this the current recommendation until this bug is resolved? Thanks!
        Hide
        wohali Joan Touzet added a comment -

        I know it's a bit late, but since this issue still seems to occur, I have a PR open with Adam Kocoloski's hibernate tweak against 2.0.0: https://github.com/apache/couchdb/pull/510

        Show
        wohali Joan Touzet added a comment - I know it's a bit late, but since this issue still seems to occur, I have a PR open with Adam Kocoloski 's hibernate tweak against 2.0.0: https://github.com/apache/couchdb/pull/510
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit 7c3aef6999691bd02fbae56a72c991a6150af261 in couchdb's branch refs/heads/master from Joan Touzet
        [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=7c3aef6 ]

        Hibernate couch_stream after each write

        In COUCHDB-1946 Adam Kocoloski investigated a memory explosion resulting
        from replication of databases with large attachments (npm fullfat). He
        was able to stabilize memory usage to a much lower level by hibernating
        couch_stream after each write. While this increases CPU utilization when
        writing attachments, it should help reduce memory utilization.

        This patch is the single change that affected a ~70% reduction in
        memory.

        No alteration to the spawn of couch_stream to change the fullsweep_after
        setting has been made, in part because this can be adjusted at the erl
        command line if desired (-erl ERL_FULLSWEEP_AFTER 0).

        Show
        jira-bot ASF subversion and git services added a comment - Commit 7c3aef6999691bd02fbae56a72c991a6150af261 in couchdb's branch refs/heads/master from Joan Touzet [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=7c3aef6 ] Hibernate couch_stream after each write In COUCHDB-1946 Adam Kocoloski investigated a memory explosion resulting from replication of databases with large attachments (npm fullfat). He was able to stabilize memory usage to a much lower level by hibernating couch_stream after each write. While this increases CPU utilization when writing attachments, it should help reduce memory utilization. This patch is the single change that affected a ~70% reduction in memory. No alteration to the spawn of couch_stream to change the fullsweep_after setting has been made, in part because this can be adjusted at the erl command line if desired (-erl ERL_FULLSWEEP_AFTER 0).
        Hide
        jira-bot ASF subversion and git services added a comment -

        Commit f073391aece76bbd0f90d7f74bb548753897a465 in couchdb's branch refs/heads/1.6.x from Joan Touzet
        [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=f073391 ]

        Hibernate couch_stream after each write

        In COUCHDB-1946 Adam Kocoloski investigated a memory explosion resulting
        from replication of databases with large attachments (npm fullfat). He
        was able to stabilize memory usage to a much lower level by hibernating
        couch_stream after each write. While this increases CPU utilization when
        writing attachments, it should help reduce memory utilization.

        This patch is the single change that affected a ~70% reduction in
        memory.

        No alteration to the spawn of couch_stream to change the fullsweep_after
        setting has been made, in part because this can be adjusted at the erl
        command line if desired (-erl ERL_FULLSWEEP_AFTER 0).

        +1 for 2.0.0 and 1.6.x from @davisp, see #510 for details.

        Show
        jira-bot ASF subversion and git services added a comment - Commit f073391aece76bbd0f90d7f74bb548753897a465 in couchdb's branch refs/heads/1.6.x from Joan Touzet [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=f073391 ] Hibernate couch_stream after each write In COUCHDB-1946 Adam Kocoloski investigated a memory explosion resulting from replication of databases with large attachments (npm fullfat). He was able to stabilize memory usage to a much lower level by hibernating couch_stream after each write. While this increases CPU utilization when writing attachments, it should help reduce memory utilization. This patch is the single change that affected a ~70% reduction in memory. No alteration to the spawn of couch_stream to change the fullsweep_after setting has been made, in part because this can be adjusted at the erl command line if desired (-erl ERL_FULLSWEEP_AFTER 0). +1 for 2.0.0 and 1.6.x from @davisp, see #510 for details.
        Hide
        wohali Joan Touzet added a comment -

        After years of inactivity I've decided to merge Adam Kocoloski's patch. This will increase CPU utilisation when writing attachments but should drastically reduce memory requirements on the target couch when replicating large attachments. If further memory reduction is required use of -erl ERL_FULLSWEEP_AFTER 0 in the launch command is recommended.

        Show
        wohali Joan Touzet added a comment - After years of inactivity I've decided to merge Adam Kocoloski 's patch. This will increase CPU utilisation when writing attachments but should drastically reduce memory requirements on the target couch when replicating large attachments. If further memory reduction is required use of -erl ERL_FULLSWEEP_AFTER 0 in the launch command is recommended.
        Hide
        kocolosk Adam Kocoloski added a comment -

        Show
        kocolosk Adam Kocoloski added a comment -

          People

          • Assignee:
            wohali Joan Touzet
            Reporter:
            stelcheck Marc Trudel
          • Votes:
            1 Vote for this issue
            Watchers:
            15 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development