[COUCHDB-2484] replication crashes

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Auto Closed
• Affects Version/s: 1.x.x
• Component/s: Database Core
• Assignee: Unassigned
• Reporter: Gunther Gruber
• Created: 01/Dec/14 13:09
• Updated: 06/Oct/17 16:55
• Resolved: 06/Oct/17 16:55

Description

We are Using Couchdb Version 1.2.0 with 8.3T of data, biggest Database ist 2.1T. At this moment we switch to new hardware with more storage space. We copied the files with rsync and started the replication.

One system is already in sync, the other is doing the replication.

I appreciate that besides the errors in the log, the first system is now in sync.

The log looks like the following

Retrying POST request to http://replication:XXXX/database/_revs_diff in 0.5 seconds due to error req_timedout

and then

Mon, 01 Dec 2014 13:00:28 GMT] [error] [<0.27044.1>] ** Generic server <0.27044.1> terminating

• • Last message in was {'EXIT',<0.26965.1>,killed}
  • When Server state == {state,<0.26965.1>,<0.27045.1>,40,
    {httpdb,
    "http://replication:XXX@XXX.5984/sm_chemie/",
    nil,
    [ {"Accept","application/json"},
    {"User-Agent","CouchDB/1.2.0"}],
    30000,
    [{socket_options,
    [{recbuf,262144},
    {sndbuf,262144},
    {nodelay,true},
    {keepalive,true}]}],
    10,250,<0.26966.1>,40},
    {httpdb,
    "http://replication:XXX@XXX:5984/sm_chemie/",
    nil,
    [{"Accept","application/json"}

    ,
    

    {"User-Agent","CouchDB/1.2.0"}

    ],
    30000,
    [

    Unknown macro: {socket_options, [{recbuf,262144}, {sndbuf,262144}, {nodelay,true}, {keepalive,true}]}

    ],
    10,250,<0.26968.1>,40},
    [],nil,nil,nil,
    

    {rep_stats,0,0,0,0,0}

    ,
    nil,nil,
    

    Unknown macro: {batch,[],0}

    }

  • Reason for termination ==
  • killed

[Mon, 01 Dec 2014 13:00:28 GMT] [error] [<0.27042.1>] {error_report,<0.31.0>,
{<0.27042.1>,crash_report,
[[{initial_call,
{couch_replicator_worker,init,['Argument__1']}},

{pid,<0.27042.1>}

,

{registered_name,[]}

,
{error_info,
{exit,killed,
[{gen_server,terminate,6,
[

{file,"gen_server.erl"}

,

{line,747}

]},
{proc_lib,init_p_do_apply,3,
[

{file,"proc_lib.erl"}

,

{line,227}

]}]}},

{ancestors, [<0.26965.1>,couch_rep_sup,couch_primary_services, couch_server_sup,<0.32.0>]}

,

{messages,[]}

,

{links,[<0.27043.1>]}

,
{dictionary,
[{last_stats_report,{1417,438797,704976}}]},

{trap_exit,true}

,

{status,running}

,

{heap_size,377}

,

{stack_size,24}

,

{reductions,372}

],
[]]}}

It seems to me like a timeout and the replication task then exits. I allready played arround with the configuration setting with no succes. I can provide more information if needed.

/etc/couchdb/local.d/001-user_config.ini
[couchdb]
file_compression = snappy
max_dbs_open = 400

[httpd]
bind_address = ::
server_options = [

{backlog, 128}

,

{acceptor_pool_size, 16}

]
socket_options = [

{recbuf, 262144}, {sndbuf, 262144}, {nodelay, true}, {keepalive, true}]

[couch_httpd_auth]
secret =

[log_level_by_module]
couch_httpd = warning
couch_replicator = debug
couch_query_servers = warning

[daemons]
httpsd = {couch_httpd, start_link, [https]}

[ssl]
cert_file = /etc/couchdb/ssl/certs/couchdb-couch1.prime.adns.de.pem
key_file = /etc/couchdb/ssl/private/couchdb-couch1.prime.adns.de.pem
verify_ssl_certificates = false

[replicator]
worker_batch_size = 2000
worker_processes = 40
http_connections = 40
socket_options = [{recbuf, 262144}

,

{sndbuf, 262144}

,

{nodelay, true}

,

{keepalive, true}

]

/etc/default/couchdb

1. Sourced by init script for configuration.

COUCHDB_USER=couchdb
COUCHDB_STDOUT_FILE=/dev/null
COUCHDB_STDERR_FILE=/dev/null
COUCHDB_RESPAWN_TIMEOUT=5
COUCHDB_OPTIONS=

1. 32 Threads to handle I/O
   export ERL_FLAGS="+A 32"
2. 8192 open files
   export ERL_MAX_PORTS=8192
   ulimit -n 8192

Current solution is to restart couchdb every other hour