CouchDB
  1. CouchDB
  2. COUCHDB-1334

Indexer speedup (for non-native view servers)

    Details

    • Skill Level:
      Guru Level (Everyone buy this person a beer at the next conference!)

      Description

      The following 2 patches significantly improve view index generation/update time and reduce CPU consumption.

      The first patch makes the view updater's batching more efficient, by ensuring each btree bulk insertion adds/removes a minimum of N (=100) key/value pairts. This also makes the index file size grow not so fast with old data (old btree nodes basically). This behaviour is already done in master/trunk in the new indexer (by Paul Davis).

      The second patch maximizes the throughput with an external view server (such as couchjs). Basically it makes the pipe (erlang port) communication between the Erlang VM (couch_os_process basically) and the view server more efficient since the 2 sides spend less time block on reading from the pipe.

      Here follow some benchmarks.

      test database at http://fdmanana.iriscouch.com/test_db (1 million documents)

      branch 1.2.x

      $ echo 3 > /proc/sys/vm/drop_caches
      $ time curl http://localhost:5984/test_db/_design/test/_view/test1
      {"rows":[

      {"key":null,"value":1000000}

      ]}

      real 2m45.097s
      user 0m0.006s
      sys 0m0.007s

      view file size: 333Mb

      CPU usage:

      $ sar 1 60
      22:27:20 %usr %nice %sys %idle
      22:27:21 38 0 12 50
      (....)
      22:28:21 39 0 13 49
      Average: 39 0 13 47

      branch 1.2.x + batch patch (first patch)

      $ echo 3 > /proc/sys/vm/drop_caches
      $ time curl http://localhost:5984/test_db/_design/test/_view/test1
      {"rows":[

      {"key":null,"value":1000000}

      ]}

      real 2m12.736s
      user 0m0.006s
      sys 0m0.005s

      view file size 72Mb

      branch 1.2.x + batch patch + os_process patch

      $ echo 3 > /proc/sys/vm/drop_caches
      $ time curl http://localhost:5984/test_db/_design/test/_view/test1
      {"rows":[

      {"key":null,"value":1000000}

      ]}

      real 1m9.330s
      user 0m0.006s
      sys 0m0.004s

      view file size: 72Mb

      CPU usage:

      $ sar 1 60
      22:22:55 %usr %nice %sys %idle
      22:23:53 22 0 6 72
      (....)
      22:23:55 22 0 6 72
      Average: 22 0 7 70

      master/trunk

      $ echo 3 > /proc/sys/vm/drop_caches
      $ time curl http://localhost:5984/test_db/_design/test/_view/test1
      {"rows":[

      {"key":null,"value":1000000}

      ]}

      real 1m57.296s
      user 0m0.006s
      sys 0m0.005s

      master/trunk + os_process patch

      $ echo 3 > /proc/sys/vm/drop_caches
      $ time curl http://localhost:5984/test_db/_design/test/_view/test1
      {"rows":[

      {"key":null,"value":1000000}

      ]}

      real 0m53.768s
      user 0m0.006s
      sys 0m0.006s

        Issue Links

          Activity

            People

            • Assignee:
              Joan Touzet
              Reporter:
              Filipe Manana
            • Votes:
              2 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:

                Development