Uploaded image for project: 'CouchDB'
  1. CouchDB
  2. COUCHDB-3184

couch_mrview_compactor:recompact/1 does not handle errors in spawned process

    Details

    • Type: Bug
    • Status: Closed
    • Priority: Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Database Core
    • Labels:
      None

      Description

      The recompact function receive block only matches on tuples of the form

      {'DOWN', Ref, _, _, {updated, Pid, State2}}
      

      see: https://github.com/apache/couchdb-couch-mrview/blob/master/src/couch_mrview_compactor.erl#L156

      but errors in the spawned process running couch_index_updater:update can return different forms, as shown in these processes captured by a user below. This results in the spawning process blocking forever in the receive.

      (cloudant@vxpif-pclddb04.lmig.com)1> process_info(pid(0,24597,7370)).
      [{current_function,{couch_mrview_compactor,recompact,1}},
      {initial_call,{erlang,apply,2}},
      {status,waiting},
      {message_queue_len,1},
      {messages,[{'DOWN',#Ref<0.0.689040.41604>,process,
                          <0.9551.7382>,
                          {timeout,{gen_server,call,
                                              [couch_proc_manager,{get_proc,<<"javascript">>},5000]}}}]},
      {links,[<0.14960.3332>,<0.4951.7370>]},
      {dictionary,[{io_priority,{view_compact,<<"shards/80000000-9fffffff/default/master_party_db_perf.146124"...>>,
                                              <<"_design/sourcePrimaryKey_V">>}},
                    {task_status_props,[{changes_done,12044429},
                                        {database,<<"shards/80000000-9fffffff/default/master_party_db"...>>},
                                        {design_document,<<"_design/sourcePrimaryKey_V">>},
                                        {phase,view},
                                        {progress,79},
                                        {started_on,1471048118},
                                        {total_changes,15182867},
                                        {type,view_compaction},
                                        {updated_on,1471048883},
                                        {view,0}]},
                    {task_status_update,{{0,0,0},0}}]},
      {trap_exit,false},
      {error_handler,error_handler},
      {priority,normal},
      {group_leader,<0.237.0>},
      {total_heap_size,954562},
      {heap_size,121536},
      {stack_size,15},
      {reductions,354812127},
      {garbage_collection,[{min_bin_vheap_size,46422},
                            {min_heap_size,233},
                            {fullsweep_after,65535},
                            {minor_gcs,9}]},
      {suspending,[]}]
      
      
      (cloudant@vxpip-pclddb03.lmig.com)1> process_info(pid(0,182,3858)).
      [{current_function,{couch_mrview_compactor,recompact,1}},
      {initial_call,{erlang,apply,2}},
      {status,waiting},
      {message_queue_len,1},
      {messages,[{'DOWN',#Ref<0.0.661774.27928>,process,
                          <0.17557.3858>,
                          {{nocatch,{os_process_error,"OS process timed out."}},
                          [{couch_os_process,prompt,2,
                                              [{file,"src/couch_os_process.erl"},{line,59}]},
                            {couch_query_servers,map_doc_raw,2,
                                                [{file,"src/couch_query_servers.erl"},{line,84}]},
                            {couch_mrview_updater,'-map_docs/2-fun-0-',3,
                                                  [{file,"src/couch_mrview_updater.erl"},{line,146}]},
                            {lists,foldl,3,[{file,"lists.erl"},{line,1261}]},
                            {couch_mrview_updater,map_docs,2,
                                                  [{file,"src/couch_mrview_updater.erl"},{line,153}]}]}}]},
      {links,[<0.31690.3859>,<0.32697.3856>]},
      {dictionary,[{io_priority,{view_compact,<<"shards/40000000-5fffffff/default/master_party_db.1463838377">>,
                                              <<"_design/sourcePartyIdentifier_V">>}},
                    {task_status_props,[{changes_done,7905},
                                        {database,<<"shards/40000000-5fffffff/default/master_party_db"...>>},
                                        {design_document,<<"_design/sourcePartyIdentifier_V">>},
                                        {phase,view},
                                        {progress,0},
                                        {started_on,1471298853},
                                        {total_changes,6261085},
                                        {type,view_compaction},
                                        {updated_on,1471298856},
                                        {view,0}]},
                    {task_status_update,{{0,0,0},0}}]},
      {trap_exit,false},
      {error_handler,error_handler},
      {priority,normal},
      {group_leader,<0.218.0>},
      {total_heap_size,364609},
      {heap_size,46422},
      {stack_size,15},
      {reductions,230416},
      {garbage_collection,[{min_bin_vheap_size,46422},
                            {min_heap_size,233},
                            {fullsweep_after,65535},
                            {minor_gcs,3}]},
      {suspending,[]}]
      

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user jaydoane opened a pull request:

          https://github.com/apache/couchdb-couch-mrview/pull/58

          Retry failures in couch_mrview_compactor:recompact

          Currently, if there's an error in the spawned couch_index_updater:update
          process, the receive block will not match the message, and the calling
          process will block forever.

          This commit changes the receive pattern to match both successful updates
          as well as errors, logging the errors, and retrying the recompact a
          configurable number of times. If the configured retry count is exceeded,
          it logs an error, and returns the original state.

          COUCHDB-3184

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cloudant/couchdb-couch-mrview 3184-retry-recompact-failure

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/couchdb-couch-mrview/pull/58.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #58


          commit 1d5c6a5d92977308fd94b2b8f43463b9f2bdac71
          Author: Jay Doane <jay.s.doane@gmail.com>
          Date: 2016-10-07T20:04:13Z

          Retry failures in couch_mrview_compactor:recompact

          Currently, if there's an error in the spawned couch_index_updater:update
          process, the receive block will not match the message, and the calling
          process will block forever.

          This commit changes the receive pattern to match both successful updates
          as well as errors, logging the errors, and retrying the recompact a
          configurable number of times. If the configured retry count is exceeded,
          it logs an error, and returns the original state.

          COUCHDB-3184


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user jaydoane opened a pull request: https://github.com/apache/couchdb-couch-mrview/pull/58 Retry failures in couch_mrview_compactor:recompact Currently, if there's an error in the spawned couch_index_updater:update process, the receive block will not match the message, and the calling process will block forever. This commit changes the receive pattern to match both successful updates as well as errors, logging the errors, and retrying the recompact a configurable number of times. If the configured retry count is exceeded, it logs an error, and returns the original state. COUCHDB-3184 You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloudant/couchdb-couch-mrview 3184-retry-recompact-failure Alternatively you can review and apply these changes as the patch at: https://github.com/apache/couchdb-couch-mrview/pull/58.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #58 commit 1d5c6a5d92977308fd94b2b8f43463b9f2bdac71 Author: Jay Doane <jay.s.doane@gmail.com> Date: 2016-10-07T20:04:13Z Retry failures in couch_mrview_compactor:recompact Currently, if there's an error in the spawned couch_index_updater:update process, the receive block will not match the message, and the calling process will block forever. This commit changes the receive pattern to match both successful updates as well as errors, logging the errors, and retrying the recompact a configurable number of times. If the configured retry count is exceeded, it logs an error, and returns the original state. COUCHDB-3184
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit 1d5c6a5d92977308fd94b2b8f43463b9f2bdac71 in couchdb-couch-mrview's branch refs/heads/master from Jay Doane
          [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-mrview.git;h=1d5c6a5 ]

          Retry failures in couch_mrview_compactor:recompact

          Currently, if there's an error in the spawned couch_index_updater:update
          process, the receive block will not match the message, and the calling
          process will block forever.

          This commit changes the receive pattern to match both successful updates
          as well as errors, logging the errors, and retrying the recompact a
          configurable number of times. If the configured retry count is exceeded,
          it logs an error, and returns the original state.

          COUCHDB-3184

          Show
          jira-bot ASF subversion and git services added a comment - Commit 1d5c6a5d92977308fd94b2b8f43463b9f2bdac71 in couchdb-couch-mrview's branch refs/heads/master from Jay Doane [ https://git-wip-us.apache.org/repos/asf?p=couchdb-couch-mrview.git;h=1d5c6a5 ] Retry failures in couch_mrview_compactor:recompact Currently, if there's an error in the spawned couch_index_updater:update process, the receive block will not match the message, and the calling process will block forever. This commit changes the receive pattern to match both successful updates as well as errors, logging the errors, and retrying the recompact a configurable number of times. If the configured retry count is exceeded, it logs an error, and returns the original state. COUCHDB-3184
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/couchdb-couch-mrview/pull/58

          Show
          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/couchdb-couch-mrview/pull/58
          Show
          jaydoane Jay Doane added a comment - merged here: https://github.com/apache/couchdb-couch-mrview/commit/853c608054fd8efe5e44a699714028ca7f76c2ef

            People

            • Assignee:
              Unassigned
              Reporter:
              jaydoane Jay Doane
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development