[COUCHDB-3009] Cluster node databases unreadable when first node in cluster is down

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 2.0.0
• Component/s: Database Core
• Assignee: Unassigned
• Reporter: Jason Gordon
• Created: 03/May/16 20:27
• Updated: 02/Sep/18 07:06
• Resolved: 02/Sep/18 07:06

Description

After creating 3 nodes in a cluster. If the first node is taken down, the other two nodes' default databases (_global_changes,_metadata, _replicator, _users ) become unreadable with the error 500

{"error":"nodedown","reason":"progress not possible"}

.

Bringing up the first node, restores access. However if the first node is down, restarting nodes 2 and 3 does not restore access and also causes the user databases to become unreachable.

Note, only the first node created in the cluster causes this problem. As long as node 1 is up, nodes 2 and 3 can go up and down without having an issue.

Log messages seen on nodes 2 and 3:

15:23:46.388 [notice] cassim_metadata_cache changes listener died nocatch,{error,timeout,[{fabric_view_changes,send_changes,6,[

{file,"src/fabric_view_changes.erl"},{line,190}]},{fabric_view_changes,keep_sending_changes,8,[{file,"src/fabric_view_changes.erl"}

,

{line,82}]},{fabric_view_changes,go,5,[{file,"src/fabric_view_changes.erl"},{line,43}]}]}
15:23:46.388 [error] Error in process <0.27407.0> on node 'couchdb@198.72.252.245' with exit value:
nocatch,{error,timeout,[{fabric_view_changes,send_changes,6,[{file,"src/fabric_view_changes.erl"},{line,190}]},{fabric_view_changes,keep_sending_changes,8,[{file,"src/fabric_view_changes.erl"},{line,82}

]},{fabric_view_changes,go,5,[

{file,"src/fabric_view_changes.erl"},{line,43}]}]}

15:23:46.389 [notice] chttpd_auth_cache changes listener died nocatch,{error,timeout,[{fabric_view_changes,send_changes,6,[{file,"src/fabric_view_changes.erl"}

,

{line,190}]},{fabric_view_changes,keep_sending_changes,8,[{file,"src/fabric_view_changes.erl"},{line,82}]},{fabric_view_changes,go,5,[{file,"src/fabric_view_changes.erl"},{line,43}]}]}
15:23:46.389 [error] Error in process <0.27414.0> on node 'couchdb@198.72.252.245' with exit value:
nocatch,{error,timeout,[{fabric_view_changes,send_changes,6,[{file,"src/fabric_view_changes.erl"},{line,190}

]},{fabric_view_changes,keep_sending_changes,8,[

{file,"src/fabric_view_changes.erl"},{line,82}]},{fabric_view_changes,go,5,[{file,"src/fabric_view_changes.erl"}

,

{line,43}

]}]}

15:23:51.391 [error] gen_server chttpd_auth_cache terminated with reason: no case clause matching

{error,read_failure} in chttpd_auth_cache:ensure_auth_ddoc_exists/2 line 187
15:23:51.391 [error] CRASH REPORT Process chttpd_auth_cache with 1 neighbours exited with reason: no case clause matching {error,read_failure}

in chttpd_auth_cache:ensure_auth_ddoc_exists/2 line 187 in gen_server:terminate/7 line 826
15:23:51.391 [error] Supervisor chttpd_sup had child undefined started with chttpd_auth_cache:start_link() at <0.27413.0> exit with reason no case clause matching

{error,read_failure}

in chttpd_auth_cache:ensure_auth_ddoc_exists/2 line 187 in context child_terminated