CouchDB
  1. CouchDB
  2. COUCHDB-976

Improve database read and write performance using 2 couch_files

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Right now we use a single couch_file server for both updating a database and reading from a database.
      This is a contention point, as concurrent read/write access to a database implies having processes waiting for access to the couch_file server.

      The following patches add a couch_file server that is used only by the DB updater process and another couch_file meant to be used by anyone else only for read operations:

      https://github.com/fdmanana/couchdb/compare/updater_dedicated_fd

      Some performance measurements:

      1. updater_fd vs trunk (small docs, 1Kb each)

      $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
      -name1 updater_fd_small_docs -name2 trunk \
      -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
      --duration 300

      http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a616

      1. updater_fd vs trunk (large docs, 100Kb each)

      $ node tests/compare_write_and_read.js --wclients 50 --rclients 200 \
      -name1 updater_fd_large_docs -name2 trunk \
      -url1 http://localhost:5984/ -url2 http://localhost:5985/ \
      --duration 300 --doc large

      http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104a7a7

      We can see that both the response time and throughput gets significantly better for both read and writes.

      If no objections I'll commit it to trunk.

        Activity

        Filipe Manana created issue -
        Hide
        Randall Leeds added a comment -

        Wonderful!!

        Your results confirm all my expectations about this patch.
        It looks great to me.

        I would keep your ets-based cache patch in mind and be sure that it will be easy enough to share one cache even with two FDs, but otherwise I have zero concerns.

        Show
        Randall Leeds added a comment - Wonderful!! Your results confirm all my expectations about this patch. It looks great to me. I would keep your ets-based cache patch in mind and be sure that it will be easy enough to share one cache even with two FDs, but otherwise I have zero concerns.
        Hide
        Randall Leeds added a comment -

        Actually, I read the graphs wrong.

        I keep accidentally comparing the reads to the writes thinking I'm comparing trunk to the patch. Oops.
        Can't actually tell if it's significantly better, but I'll try to run some tests myself because I expect it to be.

        I think the difference will be more dramatic when you crank the number of clients up much higher.

        Show
        Randall Leeds added a comment - Actually, I read the graphs wrong. I keep accidentally comparing the reads to the writes thinking I'm comparing trunk to the patch. Oops. Can't actually tell if it's significantly better, but I'll try to run some tests myself because I expect it to be. I think the difference will be more dramatic when you crank the number of clients up much higher.
        Hide
        Filipe Manana added a comment -

        No significant gains?

        For small docs:

        • for writes I see in average response times reduced by 100 to 200ms;
        • for reads I see in average response times reduced by 100ms.

        For large docs:

        • writes get in average the response time reduced by 1s to 1.5s;
        • idem for reads

        And the cache patch forget it, I don't think it's worth it, perhaps only for large docs (100kb), however it doesn't boost performance as much as this patch. Its gains are generally smaller than 5ms, which is almost insignificant, not to mention the additional complexity it brings: more code, more gen_servers, 1 ets per cache, etc.
        I have an analysis of the caching at https://issues.apache.org/jira/browse/COUCHDB-913

        Show
        Filipe Manana added a comment - No significant gains? For small docs: for writes I see in average response times reduced by 100 to 200ms; for reads I see in average response times reduced by 100ms. For large docs: writes get in average the response time reduced by 1s to 1.5s; idem for reads And the cache patch forget it, I don't think it's worth it, perhaps only for large docs (100kb), however it doesn't boost performance as much as this patch. Its gains are generally smaller than 5ms, which is almost insignificant, not to mention the additional complexity it brings: more code, more gen_servers, 1 ets per cache, etc. I have an analysis of the caching at https://issues.apache.org/jira/browse/COUCHDB-913
        Hide
        Randall Leeds added a comment -

        Yeah, you're right. The gains are there.

        As for the caching, I think maybe you're right that we should leave it out. It was a workaround for the same problem this addresses: too many reads blocking the write path on couch_file. Problem solved. Great work, Filipe. Thanks for doing this patch.

        Show
        Randall Leeds added a comment - Yeah, you're right. The gains are there. As for the caching, I think maybe you're right that we should leave it out. It was a workaround for the same problem this addresses: too many reads blocking the write path on couch_file. Problem solved. Great work, Filipe. Thanks for doing this patch.
        Hide
        Filipe Manana added a comment -

        Thanks.

        A relaximation test for 2 minutes (instead of 5) helps out verifying the gains, since point samples are not so close to each other:

        http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104b99e

        regards

        Show
        Filipe Manana added a comment - Thanks. A relaximation test for 2 minutes (instead of 5) helps out verifying the gains, since point samples are not so close to each other: http://graphs.mikeal.couchone.com/#/graph/5c859b3e7d1b9bd0488cfe271104b99e regards
        Hide
        Adam Kocoloski added a comment -

        Hi Filipe, I can't see gains of the magnitude you describe in this ticket on those graphs. A 100ms improvement in reader response time for small docs? I just don't see it.

        Nevertheless, the patch is a good idea.

        Show
        Adam Kocoloski added a comment - Hi Filipe, I can't see gains of the magnitude you describe in this ticket on those graphs. A 100ms improvement in reader response time for small docs? I just don't see it. Nevertheless, the patch is a good idea.
        Hide
        Filipe Manana added a comment -

        Adam,

        Look when T is:

        • between 50 000 and 75 000, most points have differences of about 100ms (or slightly more)
        • when T > 200 000

        In the last result I posted (the shorter one of 120secs, easier to analyse) when T > 110 000 for e.g. or when T is between 20 000 and 30 000 (between 50ms and 80ms less than trunk)

        Show
        Filipe Manana added a comment - Adam, Look when T is: between 50 000 and 75 000, most points have differences of about 100ms (or slightly more) when T > 200 000 In the last result I posted (the shorter one of 120secs, easier to analyse) when T > 110 000 for e.g. or when T is between 20 000 and 30 000 (between 50ms and 80ms less than trunk)
        Hide
        Filipe Manana added a comment -

        Committed to trunk (revision 1043352).

        Thanks everyone.

        Show
        Filipe Manana added a comment - Committed to trunk (revision 1043352). Thanks everyone.
        Filipe Manana made changes -
        Field Original Value New Value
        Status Open [ 1 ] Closed [ 6 ]
        Resolution Fixed [ 1 ]

          People

          • Assignee:
            Filipe Manana
            Reporter:
            Filipe Manana
          • Votes:
            1 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development