CouchDB
  1. CouchDB
  2. COUCHDB-1023

Batching writes of BTree nodes (when possible) and in the DB updater

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: Database Core
    • Labels:
      None

      Description

      Recently I started experimenting with batching writes in the DB updater.

      For a test of 100 writers of 1Kb documents for e.g., most often the updater collects between 20 and 30 documents to write.

      Currently it does a file:write operation for each one. Not only this is slower, but it implies more context switches and stressing the OS/filesystem by allocating few blocks very often (since we use a pure file append write mode). The same can be done in the BTree node writes.

      The following branch/patch, is an experiment of batching writes:

      https://github.com/fdmanana/couchdb/compare/batch_writes

      In couch_file there's a quick test method that compares the time taken to write X blocks of size Y versus writing a single block of size X * Y.
      Example:

      Eshell V5.8.2 (abort with ^G)
      1> Apache CouchDB 1.2.0aa777195-git (LogLevel=info) is starting.
      Apache CouchDB has started. Time to relax.
      [info] [<0.37.0>] Apache CouchDB has started on http://127.0.0.1:5984/

      1> couch_file:test(1000, 30).
      multi writes of 30 binaries, each of size 1000 bytes, took 1920us
      batch write of 30 binaries, each of size 1000 bytes, took 344us
      ok
      2>
      2> couch_file:test(4000, 30).
      multi writes of 30 binaries, each of size 4000 bytes, took 2002us
      batch write of 30 binaries, each of size 4000 bytes, took 700us
      ok
      3>

      One order of magnitude less is quite significant I would say.

      Lower response times are mostly noticeable when delayed_commits are set to true.
      Running a writes only test with this branch gave me:

      http://graphs.mikeal.couchone.com/#/graph/8bf31813eef7c0b7e37d1ea25902e544

      While with trunk I got:

      http://graphs.mikeal.couchone.com/#/graph/8bf31813eef7c0b7e37d1ea25902eb50

      These tests were done on Linux with ext4 (and OTP R14B01).

      However I'm still not 100% sure if this worth applying to trunk.
      Any thoughts?

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Filipe Manana
          • Votes:
            1 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

            • Created:
              Updated:

              Development