Details

    • Type: Bug
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Idle dbs, especially sys_dbs like _replicator shards once opened
      once for scanning would stay open forever. In a large cluster with many
      _replicator shards that can add up to a significant overhead, mostly in terms
      of number of active processes.

      Add a mechanism to close dbs which are idle.

        Issue Links

          Activity

          Hide
          githubbot ASF GitHub Bot added a comment -

          GitHub user nickva opened a pull request:

          https://github.com/apache/couchdb-couch/pull/236

          Close idle dbs

          Previously idle dbs, especially sys_dbs like _replicator shards once opened
          once for scanning would stay open forever. In a large cluster with many
          _replicator shards that can add up to a significant overhead, mostly in terms
          of number of active processes.

          Add a mechanism to close dbs which have an idle db updater. Before hibernation
          was used to limit the memory pressure, however that is often not enough in
          practice.

          Ideally timeout value would be a configuration option, however we don't want to
          add an ets call for every couch_db_updater callback, and modifiying the db
          record is prohibitive for this patch, however PSE does this work and once it
          lands we can read the idle configuration when the process starts.

          (Original idea for this belongs to Paul Davis)

          COUCHDB-3323

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/cloudant/couchdb-couch close-idle-dbs

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/couchdb-couch/pull/236.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #236


          commit 85a9f13632a6828420e01cc7d39cdaa7c19c2170
          Author: Nick Vatamaniuc <vatamane@apache.org>
          Date: 2017-03-10T19:24:01Z

          Close idle dbs

          Previously idle dbs, especially sys_dbs like _replicator shards once opened
          once for scanning would stay open forever. In a large cluster with many
          _replicator shards that can add up to a significant overhead, mostly in terms
          of number of active processes.

          Add a mechanism to close dbs which have an idle db updater. Before hibernation
          was used to limit the memory pressure, however that is often not enough in
          practice.

          Ideally timeout value would be a configuration option, however we don't want to
          add an ets call for every couch_db_updater callback, and modifiying the db
          record is prohibitive for this patch, however PSE does this work and once it
          lands we can read the idle configuration when the process starts.

          (Original idea for this belongs to Paul Davis)

          COUCHDB-3323


          Show
          githubbot ASF GitHub Bot added a comment - GitHub user nickva opened a pull request: https://github.com/apache/couchdb-couch/pull/236 Close idle dbs Previously idle dbs, especially sys_dbs like _replicator shards once opened once for scanning would stay open forever. In a large cluster with many _replicator shards that can add up to a significant overhead, mostly in terms of number of active processes. Add a mechanism to close dbs which have an idle db updater. Before hibernation was used to limit the memory pressure, however that is often not enough in practice. Ideally timeout value would be a configuration option, however we don't want to add an ets call for every couch_db_updater callback, and modifiying the db record is prohibitive for this patch, however PSE does this work and once it lands we can read the idle configuration when the process starts. (Original idea for this belongs to Paul Davis) COUCHDB-3323 You can merge this pull request into a Git repository by running: $ git pull https://github.com/cloudant/couchdb-couch close-idle-dbs Alternatively you can review and apply these changes as the patch at: https://github.com/apache/couchdb-couch/pull/236.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #236 commit 85a9f13632a6828420e01cc7d39cdaa7c19c2170 Author: Nick Vatamaniuc <vatamane@apache.org> Date: 2017-03-10T19:24:01Z Close idle dbs Previously idle dbs, especially sys_dbs like _replicator shards once opened once for scanning would stay open forever. In a large cluster with many _replicator shards that can add up to a significant overhead, mostly in terms of number of active processes. Add a mechanism to close dbs which have an idle db updater. Before hibernation was used to limit the memory pressure, however that is often not enough in practice. Ideally timeout value would be a configuration option, however we don't want to add an ets call for every couch_db_updater callback, and modifiying the db record is prohibitive for this patch, however PSE does this work and once it lands we can read the idle configuration when the process starts. (Original idea for this belongs to Paul Davis) COUCHDB-3323
          Hide
          githubbot ASF GitHub Bot added a comment -

          Github user nickva closed the pull request at:

          https://github.com/apache/couchdb-couch/pull/236

          Show
          githubbot ASF GitHub Bot added a comment - Github user nickva closed the pull request at: https://github.com/apache/couchdb-couch/pull/236
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ba0c10bcf49be66b042d8f0deaf14bcfda1a49bf in couchdb's branch refs/heads/master from Nick Vatamaniuc
          [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=ba0c10bc ]

          Close idle dbs

          Previously idle dbs, especially sys dbs like _replicator once opened
          once for scanning would stay open forever. In a large cluster with many
          _replicator shards that can add up to a significant overhead, mostly in terms
          of number of active processes.

          Add a mechanism to close dbs which have an idle db updater. Before hibernation
          was used to limit the memory pressure, however that is often not enough.

          Some databases are only read periodically so their updater would time
          out. To prevent that from happening keep the last read timestamp in
          the couch file process dictionary. Idle check then avoid closing dbs
          which have been recently read from.

          (Original idea for using timeouts in gen_server replies belongs to
          Paul Davis)

          COUCHDB-3323

          Show
          jira-bot ASF subversion and git services added a comment - Commit ba0c10bcf49be66b042d8f0deaf14bcfda1a49bf in couchdb's branch refs/heads/master from Nick Vatamaniuc [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=ba0c10bc ] Close idle dbs Previously idle dbs, especially sys dbs like _replicator once opened once for scanning would stay open forever. In a large cluster with many _replicator shards that can add up to a significant overhead, mostly in terms of number of active processes. Add a mechanism to close dbs which have an idle db updater. Before hibernation was used to limit the memory pressure, however that is often not enough. Some databases are only read periodically so their updater would time out. To prevent that from happening keep the last read timestamp in the couch file process dictionary. Idle check then avoid closing dbs which have been recently read from. (Original idea for using timeouts in gen_server replies belongs to Paul Davis) COUCHDB-3323
          Hide
          jira-bot ASF subversion and git services added a comment -

          Commit ba0c10bcf49be66b042d8f0deaf14bcfda1a49bf in couchdb's branch refs/heads/COUCHDB-3298-optimize-writing-btree-nodes from Nick Vatamaniuc
          [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=ba0c10bc ]

          Close idle dbs

          Previously idle dbs, especially sys dbs like _replicator once opened
          once for scanning would stay open forever. In a large cluster with many
          _replicator shards that can add up to a significant overhead, mostly in terms
          of number of active processes.

          Add a mechanism to close dbs which have an idle db updater. Before hibernation
          was used to limit the memory pressure, however that is often not enough.

          Some databases are only read periodically so their updater would time
          out. To prevent that from happening keep the last read timestamp in
          the couch file process dictionary. Idle check then avoid closing dbs
          which have been recently read from.

          (Original idea for using timeouts in gen_server replies belongs to
          Paul Davis)

          COUCHDB-3323

          Show
          jira-bot ASF subversion and git services added a comment - Commit ba0c10bcf49be66b042d8f0deaf14bcfda1a49bf in couchdb's branch refs/heads/ COUCHDB-3298 -optimize-writing-btree-nodes from Nick Vatamaniuc [ https://gitbox.apache.org/repos/asf?p=couchdb.git;h=ba0c10bc ] Close idle dbs Previously idle dbs, especially sys dbs like _replicator once opened once for scanning would stay open forever. In a large cluster with many _replicator shards that can add up to a significant overhead, mostly in terms of number of active processes. Add a mechanism to close dbs which have an idle db updater. Before hibernation was used to limit the memory pressure, however that is often not enough. Some databases are only read periodically so their updater would time out. To prevent that from happening keep the last read timestamp in the couch file process dictionary. Idle check then avoid closing dbs which have been recently read from. (Original idea for using timeouts in gen_server replies belongs to Paul Davis) COUCHDB-3323

            People

            • Assignee:
              Unassigned
              Reporter:
              vatamane Nick Vatamaniuc
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:

                Development