CouchDB
  1. CouchDB
  2. COUCHDB-700

Performance Regression for view generation in 0.11

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: 0.11
    • Fix Version/s: 1.3
    • Component/s: Database Core
    • Labels:
      None
    • Environment:

      Ubuntu 9.04: Stock CouchDB 0.10 / Own build of CoucDB 0.11 branch.

    • Skill Level:
      Regular Contributors Level (Easy to Medium)

      Description

      Copied from mail to dev at March 15, 2010:

      I have a synthetic benchmark for view generation over 70K documents. In stock CouchDB 0.10, the view will be checkpointed about 15-17
      times. Around 9 times with the batch_save_size and batch_save_interval set to 10000. CouchDB 0.11 on the other hand performs a whopping
      108/109 checkpoints of the view. Due to shadow B-trees this generates significantly larger view files (2-3x much) and more time is spend
      writing to disk. Generating the view takes roughly twice as long in 0.11 as it does in 0.10.

      I've tracked down the problem to the new view generation architecture; particularly the small sizes of the work queues defined in couch_view_updater.erl. The attached patch decreases the number of checkpoints to around 15, and makes view generation slightly faster
      than 0.10. It basically increases the size of the write queue. Inserting a 500 ms sleep in do_writes increased the performance a bit
      more, but that is not a nice or right solution.

      I suspect the patch is not "the completely right solution (tm)", as a lot checkpoints are performed initially whereafter it backs off and
      starts to take longer time/revisions between the checkpoints. I suspect that the code is just writing repeatedly and as writes start
      to take longer time, more revisions are added per checkpoint. Though I am not really sure of this.

      Still, it is a 2 line patch, and it significantly increases view generation performance. I'd very much like to see this in 0.11, to
      avoid a rather large performance regression between 0.10 and 0.11. If 0.11 comes out as it is, we would either have to stick with 0.10 or build our own patched 0.11.

      1. min_writer_items.diff
        3 kB
        Adam Kocoloski
      2. couchdb_011_view_speedup.diff
        0.7 kB
        Henrik Thostrup Jensen

        Issue Links

          Activity

          Hide
          Henrik Thostrup Jensen added a comment -

          Increase write work queue in Couchdb. Lowers the number of writes in view generation, which leads to greater throughput / performance.

          Show
          Henrik Thostrup Jensen added a comment - Increase write work queue in Couchdb. Lowers the number of writes in view generation, which leads to greater throughput / performance.
          Hide
          Adam Kocoloski added a comment -

          Attaching a diff (probably doesn't apply) which allows a user to set the minimum number of items sent to the reducer in a single batch. I had tried to add this capability to couch_work_queue, but it proved to be much more difficult than it sounds.

          Show
          Adam Kocoloski added a comment - Attaching a diff (probably doesn't apply) which allows a user to set the minimum number of items sent to the reducer in a single batch. I had tried to add this capability to couch_work_queue, but it proved to be much more difficult than it sounds.
          Hide
          Adam Kocoloski added a comment -

          Fixed in COUCHDB-1334

          Show
          Adam Kocoloski added a comment - Fixed in COUCHDB-1334

            People

            • Assignee:
              Unassigned
              Reporter:
              Henrik Thostrup Jensen
            • Votes:
              0 Vote for this issue
              Watchers:
              0 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development