[COUCHDB-1212] Newly created user accounts cannot sign-in after _user database crashes

• Type: Bug
• Status: Closed
• Priority: Critical
• Resolution: Won't Fix
• Affects Version/s: 1.0.2
• Component/s: Database Core, HTTP Interface
• Labels: _users authentication

Environment

Ubuntu 10.10, Erlang R14B02 (erts-5.8.3)

• Skill Level: Dont Know
• Assignee: Unassigned
• Reporter: Jan van den Berg
• Created: 03/Jul/11 15:16
• Updated: 02/Sep/18 06:33
• Resolved: 02/Sep/18 06:33

Description

We have one (4,5 GB) couch database and we use the (default) _users database to store user accounts for a website. Once a week we need to restart couchdb because newly sign-up user accounts cannot login any more. They get a HTTP statuscode 401 from the _session HTTP interface. We update, and compact the database three times a day.

This is the a stacktrace I see in the couch database log prior to when these issues occur.

----------- couch.log ---------------

[Wed, 29 Jun 2011 22:02:46 GMT] [info] [<0.117.0>] Starting compaction for db "fbm"

[Wed, 29 Jun 2011 22:02:46 GMT] [info] [<0.5753.79>] 127.0.0.1 - - 'POST' /fbm/_compact 202

[Wed, 29 Jun 2011 22:02:46 GMT] [info] [<0.5770.79>] 127.0.0.1 - - 'POST' /fbm/_view_cleanup 202

[Wed, 29 Jun 2011 22:10:19 GMT] [info] [<0.5773.79>] 86.9.246.184 - - 'GET' /_session 200

[Wed, 29 Jun 2011 22:24:39 GMT] [info] [<0.6236.79>] 85.28.105.161 - - 'GET' /_session 200

[Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.84.0>] ** Generic server couch_server terminating

• • Last message in was {open,<<"fbm">>,
    [{user_ctx,{user_ctx,null,[],undefined}}]}
  • When Server state ==
    Unknown macro: {server,"/opt/couchbase-server/var/lib/couchdb", {re_pattern,0,0, <<69,82,67,80,116,0,0,0,16,0,0,0,1,0,0,0,0,0, 0,0,0,0,0,0,40,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 0,93,0,72,25,77,0,0,0,0,0,0,0,0,0,0,0,0,254, 255,255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0, 77,0,0,0,0,16,171,255,3,0,0,0,128,254,255, 255,7,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,69,26, 84,0,72,0>>}, 100,2,"Sat, 18 Jun 2011 14}
  • Reason for termination ==
  • {timeout,{gen_server,call,[<0.116.0>, {open_ref_count,<0.10417.79>}]}}
    
    
    [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.84.0>] {error_report,<0.31.0>,
    {<0.84.0>,crash_report,
    [[{initial_call,{couch_server,init,['Argument__1']}},
    {pid,<0.84.0>},
    {registered_name,couch_server},
    {error_info,
    {exit,
    {timeout,
    {gen_server,call,
    [<0.116.0>,{open_ref_count,<0.10417.79>}

    ]}},
    [

    {gen_server,terminate,6},{proc_lib,init_p_do_apply,3}]}},
    {ancestors,[couch_primary_services,couch_server_sup,<0.32.0>]},
    {messages,[]},
    {links,[<0.91.0>,<0.483.0>,<0.116.0>,<0.79.0>]},
    {dictionary,[]},
    {trap_exit,true},
    {status,running},
    {heap_size,6765},
    {stack_size,24},
    {reductions,206710598}],
    []]}}
    [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.79.0>] {error_report,<0.31.0>,
    {<0.79.0>,supervisor_report,
    [{supervisor,{local,couch_primary_services}},
    {errorContext,child_terminated},
    {reason,
    {timeout,
    {gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}}},
    {offender,
    [{pid,<0.84.0>},
    {name,couch_server},
    {mfargs,{couch_server,sup_start_link,[]}},
    {restart_type,permanent},
    {shutdown,1000},
    {child_type,worker}]}]}}
    
    [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.91.0>] ** Generic server <0.91.0> terminating
    ** Last message in was {'EXIT',<0.84.0>,
    {timeout,
    {gen_server,call,
    [<0.116.0>,
    {open_ref_count,<0.10417.79>}]}}}
    ** When Server state == {db,<0.91.0>,<0.92.0>,nil,<<"1308405644393791">>,
    <0.90.0>,<0.94.0>,
    {db_header,5,91,0,
    {378285,{30,9}},
    {380466,39},
    nil,0,nil,nil,1000},
    91,
    {btree,<0.90.0>,
    {378285,{30,9}},
    #Fun<couch_db_updater.7.10053969>,
    #Fun<couch_db_updater.8.35220795>,
    #Fun<couch_btree.5.124754102>,
    #Fun<couch_db_updater.9.107593676>},
    {btree,<0.90.0>,
    {380466,39},
    #Fun<couch_db_updater.10.30996817>,
    #Fun<couch_db_updater.11.96515267>,
    #Fun<couch_btree.5.124754102>,
    #Fun<couch_db_updater.12.117826253>},
    {btree,<0.90.0>,nil,#Fun<couch_btree.0.83553141>, #Fun<couch_btree.1.30790806>, #Fun<couch_btree.2.124754102>,nil},
    91,<<"_users">>,
    "/opt/couchbase-server/var/lib/couchdb/_users.couch",
    Fun<couch_doc.7.50754398>,
    [],nil,
    {user_ctx,null,[],undefined},
    nil,1000,
    [before_header,after_header,on_file_open],
    true}
    ** Reason for termination ==
    ** {timeout,{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}}
    
    
    [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.91.0>] {error_report,<0.31.0>,
    {<0.91.0>,crash_report,
    [[{initial_call,{couch_db,init,['Argument__1']}},
    {pid,<0.91.0>},
    {registered_name,[]},
    {error_info,
    {exit,
    {timeout,
    {gen_server,call,
    [<0.116.0>,{open_ref_count,<0.10417.79>}]}},
    [{gen_server,terminate,6}

    ,

    {proc_lib,init_p_do_apply,3}]}},
    {ancestors,[<0.89.0>]},
    {messages,[]},
    {links,[]},
    {dictionary,[]},
    {trap_exit,true},
    {status,running},
    {heap_size,610},
    {stack_size,24},
    {reductions,8797798}],
    []]}}
    
    [Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.300.0>] Shutting down view group server, monitored db is closing.
    
    [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.10417.79>] Uncaught error in HTTP request: {exit,
    {{timeout,
    {gen_server,call,
    [<0.116.0>,
    {open_ref_count,<0.10417.79>}]}},
    {gen_server,call,
    [couch_server,
    {open,<<"fbm">>,
    [{user_ctx,
    {user_ctx,null,[],undefined}}]},
    infinity]}}}
    
    [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.483.0>] ** Generic server <0.483.0> terminating
    ** Last message in was {'EXIT',<0.84.0>,
    {timeout,
    {gen_server,call,
    [<0.116.0>,
    {open_ref_count,<0.10417.79>}]}}}
    ** When Server state == {db,<0.483.0>,<0.484.0>,nil,<<"1308405937993370">>,
    <0.4643.19>,<0.4645.19>,
    {db_header,5,890453,0,
    {3279126950,{752003,0}},
    {3279118313,752003},
    {3279132318,[]},
    0,nil,3279127184,1000},
    890453,
    {btree,<0.4643.19>,
    {3279126950,{752003,0}},
    #Fun<couch_db_updater.7.10053969>,
    #Fun<couch_db_updater.8.35220795>,
    #Fun<couch_btree.5.124754102>,
    #Fun<couch_db_updater.9.107593676>},
    {btree,<0.4643.19>,
    {3279118313,752003},
    #Fun<couch_db_updater.10.30996817>,
    #Fun<couch_db_updater.11.96515267>,
    #Fun<couch_btree.5.124754102>,
    #Fun<couch_db_updater.12.117826253>},
    {btree,<0.4643.19>,
    {3279132318,[]},
    #Fun<couch_btree.0.83553141>,
    #Fun<couch_btree.1.30790806>,
    #Fun<couch_btree.2.124754102>,nil},
    890453,<<"fbm_full">>,
    "/opt/couchbase-server/var/lib/couchdb/fbm_full.couch",
    Fun<couch_doc.7.50754398>,
    [{<<"admins">>,
    {[{<<"names">>,[]},
    {<<"roles">>,[<<"import">>]}]}},
    {<<"readers">>,
    {[{<<"names">>,[]},{<<"roles">>,[]}]}}],
    3279127184,
    {user_ctx,null,[],undefined},
    nil,1000,
    [before_header,after_header,on_file_open],
    false}
    ** Reason for termination ==
    ** {timeout,{gen_server,call,[<0.116.0>,{open_ref_count,<0.10417.79>}]}}
    
    
    [Wed, 29 Jun 2011 22:25:06 GMT] [error] [<0.483.0>] {error_report,<0.31.0>,
    {<0.483.0>,crash_report,
    [[{initial_call,{couch_db,init,['Argument__1']}},
    {pid,<0.483.0>},
    {registered_name,[]},
    {error_info,
    {exit,
    {timeout,
    {gen_server,call,
    [<0.116.0>,{open_ref_count,<0.10417.79>}]}},
    [{gen_server,terminate,6},{proc_lib,init_p_do_apply,3}

    ]}},
    

    {ancestors,[<0.480.0>]}

    ,
    

    {messages,[]}

    ,
    

    {links,[]}

    ,
    

    {dictionary,[]}

    ,
    

    {trap_exit,true}

    ,
    

    {status,running}

    ,
    

    {heap_size,6765}

    ,
    

    {stack_size,24}

    ,
    

    {reductions,1389}

    ],
    []]}}

[Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.2984.19>] Shutting down view group server, monitored db is closing.

[Wed, 29 Jun 2011 22:25:06 GMT] [info] [<0.10417.79>] Stacktrace: [

{gen_server,call,3}

,

{couch_server,open,2}

,

{couch_db,open,2}

,

{couch_httpd_db,do_db_req,2}

,

{couch_httpd,handle_request_int,5}

,

{mochiweb_http,headers,5}

,

{proc_lib,init_p_do_apply,3}

]
------- end --------

Here's the log file of me signing in as an admin, creating a new user, and trying to sign-in as the newly created user.

------ couch.log -------
[Fri, 01 Jul 2011 18:37:16 GMT] [info] [<0.20439.91>] 93.92.103.118 - - 'POST' /_session 200

[Fri, 01 Jul 2011 18:37:16 GMT] [info] [<0.20457.91>] checkpointing view update at seq 91 for _users _design/_auth

[Fri, 01 Jul 2011 18:37:16 GMT] [info] [<0.20439.91>] 93.92.103.118 - - 'GET' /_users/_design/_auth/_list/secure/users 200

[Fri, 01 Jul 2011 18:38:35 GMT] [info] [<0.20456.91>] 93.92.103.118 - - 'PUT' /_users/org.couchdb.user:[e-mail] 201

[Fri, 01 Jul 2011 18:38:35 GMT] [info] [<0.20457.91>] checkpointing view update at seq 92 for _users _design/_auth

[Fri, 01 Jul 2011 18:38:35 GMT] [info] [<0.20456.91>] 93.92.103.118 - - 'GET' /_users/_design/_auth/_list/secure/users 200

[Fri, 01 Jul 2011 18:38:47 GMT] [info] [<0.20456.91>] 93.92.103.118 - - 'GET' /_users/_design/_auth/_list/secure/users?key=%22org.couchdb.user:[e-mail]%22 200

[Fri, 01 Jul 2011 18:38:47 GMT] [info] [<0.20456.91>] 93.92.103.118 - - 'PUT' /_users/org.couchdb.user:[e-mail] 201

[Fri, 01 Jul 2011 18:39:00 GMT] [info] [<0.20547.91>] 93.92.103.118 - - 'GET' /_session 200

[Fri, 01 Jul 2011 18:39:01 GMT] [info] [<0.20547.91>] 93.92.103.118 - - 'GET' /fbm/_design/api/_list/secure/competitions 200

[Fri, 01 Jul 2011 18:39:12 GMT] [info] [<0.20547.91>] 93.92.103.118 - - 'POST' /_session 401

[Fri, 01 Jul 2011 18:39:22 GMT] [info] [<0.20547.91>] 93.92.103.118 - - 'POST' /_session 401

------- end ---------