[COUCHDB-2426] Timeouts in couch_os_process:start_link/1 leave dangling couch_query_servers

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.0.0
• Fix Version/s: 2.0.0
• Component/s: JavaScript View Server
• Assignee: Unassigned
• Reporter: Mike Wallace
• Created: 31/Oct/14 19:36
• Updated: 27/Oct/15 21:50
• Resolved: 23/Jan/15 16:07

Description

Similar to COUCHDB-2425, if the couch_os_process:start_link/1 call made in couch_proc_manager:new_proc_int/1 times out [1] then a reply is not returned to the calling process. When the caller is couch_query_servers:get_os_process/1 [4] there is an infinite timeout so that process will hang around until either the node reboots or someone intervenes.

The user-facing symptom is an entry in _active_tasks that makes no progress and never goes away.

The easiest way to reproduce this is to create a new view and then make couch_os_process time out by patching an unreasonable timeout, e.g.: (assuming a live dev/run instance):

1. Create DB, add a ddoc and a doc:

$ curl -X PUT http://localhost:15984/kitteh
{"ok":true}
$ curl -X POST http://localhost:15984/kitteh -d '{"_id":"_design/view", "views": {"test": {"map": "function(doc) { emit(doc.id, 1); }"}}}' -H 'Content-Type: application/json'
{"ok":true,"id":"_design/view","rev":"1-ef10f980522d4c8e691e3c26d4c3fac5"}
$ curl -X PUT http://localhost:15984/kitteh/ohai -d '{}'
{"ok":true,"id":"ohai","rev":"1-967a00dff5e02add41819138abb3284d"}

2. Apply https://gist.github.com/mikewallace1979/aca1bc331e5eed693357 then re-run make and dev/run.

3. Attempt to query the view:

$ curl -X GET http://localhost:15984/kitteh/_design/view/_view/test
{"error":"timeout","reason":"The request could not be processed in a reasonable amount of time."}

4. Observe index build tasks which never go away:

$ curl -X GET http://localhost:15986/_active_tasks
[{"pid":"<0.2527.0>","changes_done":0,"database":"shards/40000000-5fffffff/kitteh.1414783387","design_document":"_design/view","progress":0,"started_on":1414783454,"total_changes":1,"type":"indexer","updated_on":1414783454},{"pid":"<0.2553.0>","changes_done":0,"database":"shards/00000000-1fffffff/kitteh.1414783387","design_document":"_design/view","progress":0,"started_on":1414783454,"total_changes":1,"type":"indexer","updated_on":1414783454}]

[1] https://github.com/apache/couchdb-couch/blob/master/src/couch_proc_manager.erl#L401