Jackrabbit Oak
  1. Jackrabbit Oak
  2. OAK-1153

Write test to measure throughput on Mongo with multiple reader and writer

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19
    • Component/s: mongomk
    • Labels:
      None

      Description

      To check how raw access to Mongo performs with multiple reader and writers we need to have some throughput test.

      It would also server to decide if writing blob in separate databse provides any benefit or not. Refer to [1] for discussion around this concern

      [1] http://markmail.org/thread/aukqziaq3hryvtaz

        Activity

        Hide
        Alex Parvulescu added a comment -

        Bulk close for the 0.19 release.

        Show
        Alex Parvulescu added a comment - Bulk close for the 0.19 release.
        Hide
        Chetan Mehrotra added a comment -

        Planned work done for now

        Show
        Chetan Mehrotra added a comment - Planned work done for now
        Hide
        Shashank Gupta added a comment - - edited

        Write throughput comparision between sharded writers and non-sharded writes. Throughput slightly increased over non-sharded writes.

        • Mongo cluster with three shards. Each shard is single primary mongod instance.
        • Non-sharded write throughput.
          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 1 32 137 1272 120 1 22.2 MB NORMAL
          true true 1 32 346 914 120 2 16.0 MB JOURNAL_SAFE
          true true 1 32 381 952 120 3 16.6 MB FSYNC_SAFE
          true true 1 32 207 1033 120 1 18.1 MB SAFE
          true true 1 32 188 926 120 1 16.2 MB NONE
        • Sharded write throughput
          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 1 32 437 1600 120 3 28.0 MB NORMAL
          true true 1 32 612 1056 120 5 18.5 MB JOURNAL_SAFE
          true true 1 32 625 1025 120 5 17.9 MB FSYNC_SAFE
          true true 1 32 375 966 120 3 16.9 MB SAFE
          true true 1 32 281 1465 120 2 25.6 MB NONE
        Show
        Shashank Gupta added a comment - - edited Write throughput comparision between sharded writers and non-sharded writes. Throughput slightly increased over non-sharded writes. Mongo cluster with three shards. Each shard is single primary mongod instance. Non-sharded write throughput. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 137 1272 120 1 22.2 MB NORMAL true true 1 32 346 914 120 2 16.0 MB JOURNAL_SAFE true true 1 32 381 952 120 3 16.6 MB FSYNC_SAFE true true 1 32 207 1033 120 1 18.1 MB SAFE true true 1 32 188 926 120 1 16.2 MB NONE Sharded write throughput remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 437 1600 120 3 28.0 MB NORMAL true true 1 32 612 1056 120 5 18.5 MB JOURNAL_SAFE true true 1 32 625 1025 120 5 17.9 MB FSYNC_SAFE true true 1 32 375 966 120 3 16.9 MB SAFE true true 1 32 281 1465 120 2 25.6 MB NONE
        Hide
        Shashank Gupta added a comment - - edited

        Throughput comparision in Amazon AWS. Throughput dropped considerably( nearly 70%) in replica set cluster.

        • All nodes running seperately on m1.medium in availbility zone us-east-1a.
        • All nodes using instance storage
        • Max wait time increased to 240s from default 120s
        • Remote standalone mongodb instance.
          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 1 32 40 885 60 0 30.9 MB NORMAL
          true true 1 32 59 787 60 0 27.5 MB JOURNAL_SAFE
          true true 1 32 49 788 60 0 27.5 MB FSYNC_SAFE
          true true 1 32 32 848 60 0 29.6 MB SAFE
          true true 1 32 61 845 60 1 29.5 MB NONE
        • Three member replica set with primary + 2 secondary. Each node is running on seperate ec2 instance.
          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 1 32 1 324 60 0 11.3 MB NORMAL
          true true 1 32 12 229 60 0 8.0 MB MAJORITY
          true true 1 32 14 268 60 0 9.4 MB JOURNAL_SAFE
          true true 1 32 13 262 60 0 9.2 MB FSYNC_SAFE
          true true 1 32 9 273 60 0 9.5 MB SAFE
          true true 1 32 3 328 60 0 11.5 MB NONE
          true true 1 32 9 228 60 0 8.0 MB REPLICAS_SAFE
        Show
        Shashank Gupta added a comment - - edited Throughput comparision in Amazon AWS. Throughput dropped considerably( nearly 70%) in replica set cluster. All nodes running seperately on m1.medium in availbility zone us-east-1a. All nodes using instance storage Max wait time increased to 240s from default 120s Remote standalone mongodb instance. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 40 885 60 0 30.9 MB NORMAL true true 1 32 59 787 60 0 27.5 MB JOURNAL_SAFE true true 1 32 49 788 60 0 27.5 MB FSYNC_SAFE true true 1 32 32 848 60 0 29.6 MB SAFE true true 1 32 61 845 60 1 29.5 MB NONE Three member replica set with primary + 2 secondary. Each node is running on seperate ec2 instance. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 1 324 60 0 11.3 MB NORMAL true true 1 32 12 229 60 0 8.0 MB MAJORITY true true 1 32 14 268 60 0 9.4 MB JOURNAL_SAFE true true 1 32 13 262 60 0 9.2 MB FSYNC_SAFE true true 1 32 9 273 60 0 9.5 MB SAFE true true 1 32 3 328 60 0 11.5 MB NONE true true 1 32 9 228 60 0 8.0 MB REPLICAS_SAFE
        Hide
        Shashank Gupta added a comment - - edited

        Write throughput comparision between standalone vs replica set on increasing write threads. Throughput decreases from 33% to 75%.

        • Remote standalone mongodb instance.
          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 1 4 508 31 5 101 13.0 MB SAFE
          true true 1 8 290 32 5 58 13.4 MB SAFE
          true true 1 16 5 39 5 1 16.4 MB SAFE
          true true 1 32 4 54 5 0 22.6 MB SAFE
        • Three member replica set with primary| secondary and arbiter running on single remote machine.
          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 1 4 70 7 5 14 2.9 MB SAFE
          true true 1 8 80 13 5 16 5.5 MB SAFE
          true true 1 16 4 20 5 0 8.4 MB SAFE
          true true 1 32 5 38 5 1 15.9 MB SAFE
        Show
        Shashank Gupta added a comment - - edited Write throughput comparision between standalone vs replica set on increasing write threads. Throughput decreases from 33% to 75%. Remote standalone mongodb instance. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 4 508 31 5 101 13.0 MB SAFE true true 1 8 290 32 5 58 13.4 MB SAFE true true 1 16 5 39 5 1 16.4 MB SAFE true true 1 32 4 54 5 0 22.6 MB SAFE Three member replica set with primary| secondary and arbiter running on single remote machine. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 4 70 7 5 14 2.9 MB SAFE true true 1 8 80 13 5 16 5.5 MB SAFE true true 1 16 4 20 5 0 8.4 MB SAFE true true 1 32 5 38 5 1 15.9 MB SAFE
        Hide
        Shashank Gupta added a comment - - edited

        Write throughput decreased by approximate 33% from standalone mongodb to mongodb replica set.

        • Remote standalone mongodb instance.
          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 1 32 5 53 5 1 22.2 MB NORMAL
          true true 1 32 5 54 5 1 22.6 MB JOURNAL_SAFE
          true true 1 32 4 53 5 0 22.2 MB FSYNC_SAFE
          true true 1 32 3 54 5 0 22.6 MB SAFE
          true true 1 32 2 52 5 0 21.8 MB NONE
        • Three member replica set with primary, secondary and arbiter running on single remote machine.
          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 1 32 2 48 5 0 20.1 MB NORMAL
          true true 1 32 4 32 5 0 13.4 MB MAJORITY
          true true 1 32 4 34 5 0 14.3 MB JOURNAL_SAFE
          true true 1 32 4 35 5 0 14.7 MB FSYNC_SAFE
          true true 1 32 4 36 5 0 15.1 MB SAFE
          true true 1 32 4 33 5 0 13.8 MB REPLICAS_SAFE
          true true 1 32 5 48 5 1 20.1 MB NONE
        Show
        Shashank Gupta added a comment - - edited Write throughput decreased by approximate 33% from standalone mongodb to mongodb replica set. Remote standalone mongodb instance. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 5 53 5 1 22.2 MB NORMAL true true 1 32 5 54 5 1 22.6 MB JOURNAL_SAFE true true 1 32 4 53 5 0 22.2 MB FSYNC_SAFE true true 1 32 3 54 5 0 22.6 MB SAFE true true 1 32 2 52 5 0 21.8 MB NONE Three member replica set with primary, secondary and arbiter running on single remote machine. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 2 48 5 0 20.1 MB NORMAL true true 1 32 4 32 5 0 13.4 MB MAJORITY true true 1 32 4 34 5 0 14.3 MB JOURNAL_SAFE true true 1 32 4 35 5 0 14.7 MB FSYNC_SAFE true true 1 32 4 36 5 0 15.1 MB SAFE true true 1 32 4 33 5 0 13.8 MB REPLICAS_SAFE true true 1 32 5 48 5 1 20.1 MB NONE
        Hide
        Chetan Mehrotra added a comment -

        It might be interesting to know the performance with different write concerns (just this dimension).

        Update the test in rev 1539600. Below are results for 0 reader, 2 writer, 4 MB per insert for various write concern against a remote Mongo server

        remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
        true true 0 2 0 16 5 0 13.4 MB NORMAL
        true true 0 2 0 14 5 0 11.7 MB JOURNAL_SAFE
        true true 0 2 0 14 5 0 11.7 MB FSYNC_SAFE
        true true 0 2 0 15 5 0 12.6 MB SAFE
        true true 0 2 0 16 5 0 13.4 MB NONE
        Show
        Chetan Mehrotra added a comment - It might be interesting to know the performance with different write concerns (just this dimension). Update the test in rev 1539600 . Below are results for 0 reader, 2 writer, 4 MB per insert for various write concern against a remote Mongo server remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 0 2 0 16 5 0 13.4 MB NORMAL true true 0 2 0 14 5 0 11.7 MB JOURNAL_SAFE true true 0 2 0 14 5 0 11.7 MB FSYNC_SAFE true true 0 2 0 15 5 0 12.6 MB SAFE true true 0 2 0 16 5 0 13.4 MB NONE
        Hide
        Thomas Mueller added a comment -

        This is very interesting! Thanks a lot.

        It might be interesting to know the performance with different write concerns (just this dimension).

        > if writing blob in separate collection provides any benefit

        I think you mean "separate database"?

        Show
        Thomas Mueller added a comment - This is very interesting! Thanks a lot. It might be interesting to know the performance with different write concerns (just this dimension). > if writing blob in separate collection provides any benefit I think you mean "separate database"?
        Hide
        Chetan Mehrotra added a comment -

        Added initial test in http://svn.apache.org/r1539551

        Running the test yields following result

        remote samedb readers writers totalReads totalWrites time readThroughPut writeThroughPut
        false false 5 0 280796 0 5 56159 0 B
        false false 5 1 13312 56 5 2662 23.5 MB
        false false 5 2 3922 115 5 784 48.2 MB
        false false 5 4 1453 136 5 290 57.0 MB
        false false 10 0 392799 0 5 78559 0 B
        false false 10 1 37530 153 5 7506 64.2 MB
        false false 10 2 19955 171 5 3991 71.7 MB
        false false 10 4 13019 175 5 2603 73.4 MB
        false false 15 0 334740 0 5 66948 0 B
        false false 15 1 72733 180 5 14546 75.5 MB
        false false 15 2 40950 129 5 8190 54.1 MB
        false false 15 4 17759 189 5 3551 79.3 MB
        false false 20 0 344517 0 5 68903 0 B
        false false 20 1 93679 188 5 18735 78.9 MB
        false false 20 2 60633 192 5 12126 80.5 MB
        false false 20 4 45535 199 5 9107 83.5 MB
        false true 5 0 367334 0 5 73466 0 B
        false true 5 1 27254 36 5 5450 15.1 MB
        false true 5 2 2633 101 5 526 42.4 MB
        false true 5 4 1023 146 5 204 61.2 MB
        false true 10 0 361794 0 5 72358 0 B
        false true 10 1 39355 164 5 7871 68.8 MB
        false true 10 2 22363 179 5 4472 75.1 MB
        false true 10 4 11117 183 5 2223 76.8 MB
        false true 15 0 371182 0 5 74236 0 B
        false true 15 1 59039 164 5 11807 68.8 MB
        false true 15 2 42343 189 5 8468 79.3 MB
        false true 15 4 28134 151 5 5626 63.3 MB
        false true 20 0 353480 0 5 70696 0 B
        false true 20 1 79008 191 5 15801 80.1 MB
        false true 20 2 49861 124 5 9972 52.0 MB
        false true 20 4 30017 198 5 6003 83.0 MB
        true false 5 0 31324 0 5 6264 0 B
        true false 5 1 2763 28 5 552 11.7 MB
        true false 5 2 1313 29 5 262 12.2 MB
        true false 5 4 714 31 5 142 13.0 MB
        true false 10 0 73676 0 5 14735 0 B
        true false 10 1 5618 28 5 1123 11.7 MB
        true false 10 2 5504 28 5 1100 11.7 MB
        true false 10 4 911 30 5 182 12.6 MB
        true false 15 0 72758 0 5 14551 0 B
        true false 15 1 5644 27 5 1128 11.3 MB
        true false 15 2 3427 28 5 685 11.7 MB
        true false 15 4 1104 30 5 220 12.6 MB
        true false 20 0 74050 0 5 14810 0 B
        true false 20 1 12430 24 5 2486 10.1 MB
        true false 20 2 2888 29 5 577 12.2 MB
        true false 20 4 1253 30 5 250 12.6 MB
        true true 5 0 31143 0 5 6228 0 B
        true true 5 1 2613 28 5 522 11.7 MB
        true true 5 2 1213 29 5 242 12.2 MB
        true true 5 4 2768 26 5 553 10.9 MB
        true true 10 0 73625 0 5 14725 0 B
        true true 10 1 16300 23 5 3260 9.6 MB
        true true 10 2 2221 29 5 444 12.2 MB
        true true 10 4 865 30 5 173 12.6 MB
        true true 15 0 73177 0 5 14635 0 B
        true true 15 1 5214 27 5 1042 11.3 MB
        true true 15 2 6023 27 5 1204 11.3 MB
        true true 15 4 1113 31 5 222 13.0 MB
        true true 20 0 73911 0 5 14782 0 B
        true true 20 1 5984 27 5 1196 11.3 MB
        true true 20 2 2914 29 5 582 12.2 MB
        true true 20 4 1353 31 5 270 13.0 MB
        Show
        Chetan Mehrotra added a comment - Added initial test in http://svn.apache.org/r1539551 Running the test yields following result remote samedb readers writers totalReads totalWrites time readThroughPut writeThroughPut false false 5 0 280796 0 5 56159 0 B false false 5 1 13312 56 5 2662 23.5 MB false false 5 2 3922 115 5 784 48.2 MB false false 5 4 1453 136 5 290 57.0 MB false false 10 0 392799 0 5 78559 0 B false false 10 1 37530 153 5 7506 64.2 MB false false 10 2 19955 171 5 3991 71.7 MB false false 10 4 13019 175 5 2603 73.4 MB false false 15 0 334740 0 5 66948 0 B false false 15 1 72733 180 5 14546 75.5 MB false false 15 2 40950 129 5 8190 54.1 MB false false 15 4 17759 189 5 3551 79.3 MB false false 20 0 344517 0 5 68903 0 B false false 20 1 93679 188 5 18735 78.9 MB false false 20 2 60633 192 5 12126 80.5 MB false false 20 4 45535 199 5 9107 83.5 MB false true 5 0 367334 0 5 73466 0 B false true 5 1 27254 36 5 5450 15.1 MB false true 5 2 2633 101 5 526 42.4 MB false true 5 4 1023 146 5 204 61.2 MB false true 10 0 361794 0 5 72358 0 B false true 10 1 39355 164 5 7871 68.8 MB false true 10 2 22363 179 5 4472 75.1 MB false true 10 4 11117 183 5 2223 76.8 MB false true 15 0 371182 0 5 74236 0 B false true 15 1 59039 164 5 11807 68.8 MB false true 15 2 42343 189 5 8468 79.3 MB false true 15 4 28134 151 5 5626 63.3 MB false true 20 0 353480 0 5 70696 0 B false true 20 1 79008 191 5 15801 80.1 MB false true 20 2 49861 124 5 9972 52.0 MB false true 20 4 30017 198 5 6003 83.0 MB true false 5 0 31324 0 5 6264 0 B true false 5 1 2763 28 5 552 11.7 MB true false 5 2 1313 29 5 262 12.2 MB true false 5 4 714 31 5 142 13.0 MB true false 10 0 73676 0 5 14735 0 B true false 10 1 5618 28 5 1123 11.7 MB true false 10 2 5504 28 5 1100 11.7 MB true false 10 4 911 30 5 182 12.6 MB true false 15 0 72758 0 5 14551 0 B true false 15 1 5644 27 5 1128 11.3 MB true false 15 2 3427 28 5 685 11.7 MB true false 15 4 1104 30 5 220 12.6 MB true false 20 0 74050 0 5 14810 0 B true false 20 1 12430 24 5 2486 10.1 MB true false 20 2 2888 29 5 577 12.2 MB true false 20 4 1253 30 5 250 12.6 MB true true 5 0 31143 0 5 6228 0 B true true 5 1 2613 28 5 522 11.7 MB true true 5 2 1213 29 5 242 12.2 MB true true 5 4 2768 26 5 553 10.9 MB true true 10 0 73625 0 5 14725 0 B true true 10 1 16300 23 5 3260 9.6 MB true true 10 2 2221 29 5 444 12.2 MB true true 10 4 865 30 5 173 12.6 MB true true 15 0 73177 0 5 14635 0 B true true 15 1 5214 27 5 1042 11.3 MB true true 15 2 6023 27 5 1204 11.3 MB true true 15 4 1113 31 5 222 13.0 MB true true 20 0 73911 0 5 14782 0 B true true 20 1 5984 27 5 1196 11.3 MB true true 20 2 2914 29 5 582 12.2 MB true true 20 4 1353 31 5 270 13.0 MB

          People

          • Assignee:
            Chetan Mehrotra
            Reporter:
            Chetan Mehrotra
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development