Jackrabbit Oak
  1. Jackrabbit Oak
  2. OAK-1153

Write test to measure throughput on Mongo with multiple reader and writer

    Details

    • Type: Task Task
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.19
    • Component/s: mongomk
    • Labels:
      None

      Description

      To check how raw access to Mongo performs with multiple reader and writers we need to have some throughput test.

      It would also server to decide if writing blob in separate databse provides any benefit or not. Refer to [1] for discussion around this concern

      [1] http://markmail.org/thread/aukqziaq3hryvtaz

        Issue Links

          Activity

          Hide
          Alex Parvulescu added a comment -

          Bulk close for the 0.19 release.

          Show
          Alex Parvulescu added a comment - Bulk close for the 0.19 release.
          Hide
          Chetan Mehrotra added a comment -

          Planned work done for now

          Show
          Chetan Mehrotra added a comment - Planned work done for now
          Hide
          Shashank Gupta added a comment - - edited

          Write throughput comparision between sharded writers and non-sharded writes. Throughput slightly increased over non-sharded writes.

          • Mongo cluster with three shards. Each shard is single primary mongod instance.
          • Non-sharded write throughput.
            remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
            true true 1 32 137 1272 120 1 22.2 MB NORMAL
            true true 1 32 346 914 120 2 16.0 MB JOURNAL_SAFE
            true true 1 32 381 952 120 3 16.6 MB FSYNC_SAFE
            true true 1 32 207 1033 120 1 18.1 MB SAFE
            true true 1 32 188 926 120 1 16.2 MB NONE
          • Sharded write throughput
            remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
            true true 1 32 437 1600 120 3 28.0 MB NORMAL
            true true 1 32 612 1056 120 5 18.5 MB JOURNAL_SAFE
            true true 1 32 625 1025 120 5 17.9 MB FSYNC_SAFE
            true true 1 32 375 966 120 3 16.9 MB SAFE
            true true 1 32 281 1465 120 2 25.6 MB NONE
          Show
          Shashank Gupta added a comment - - edited Write throughput comparision between sharded writers and non-sharded writes. Throughput slightly increased over non-sharded writes. Mongo cluster with three shards. Each shard is single primary mongod instance. Non-sharded write throughput. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 137 1272 120 1 22.2 MB NORMAL true true 1 32 346 914 120 2 16.0 MB JOURNAL_SAFE true true 1 32 381 952 120 3 16.6 MB FSYNC_SAFE true true 1 32 207 1033 120 1 18.1 MB SAFE true true 1 32 188 926 120 1 16.2 MB NONE Sharded write throughput remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 437 1600 120 3 28.0 MB NORMAL true true 1 32 612 1056 120 5 18.5 MB JOURNAL_SAFE true true 1 32 625 1025 120 5 17.9 MB FSYNC_SAFE true true 1 32 375 966 120 3 16.9 MB SAFE true true 1 32 281 1465 120 2 25.6 MB NONE
          Hide
          Shashank Gupta added a comment - - edited

          Throughput comparision in Amazon AWS. Throughput dropped considerably( nearly 70%) in replica set cluster.

          • All nodes running seperately on m1.medium in availbility zone us-east-1a.
          • All nodes using instance storage
          • Max wait time increased to 240s from default 120s
          • Remote standalone mongodb instance.
            remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
            true true 1 32 40 885 60 0 30.9 MB NORMAL
            true true 1 32 59 787 60 0 27.5 MB JOURNAL_SAFE
            true true 1 32 49 788 60 0 27.5 MB FSYNC_SAFE
            true true 1 32 32 848 60 0 29.6 MB SAFE
            true true 1 32 61 845 60 1 29.5 MB NONE
          • Three member replica set with primary + 2 secondary. Each node is running on seperate ec2 instance.
            remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
            true true 1 32 1 324 60 0 11.3 MB NORMAL
            true true 1 32 12 229 60 0 8.0 MB MAJORITY
            true true 1 32 14 268 60 0 9.4 MB JOURNAL_SAFE
            true true 1 32 13 262 60 0 9.2 MB FSYNC_SAFE
            true true 1 32 9 273 60 0 9.5 MB SAFE
            true true 1 32 3 328 60 0 11.5 MB NONE
            true true 1 32 9 228 60 0 8.0 MB REPLICAS_SAFE
          Show
          Shashank Gupta added a comment - - edited Throughput comparision in Amazon AWS. Throughput dropped considerably( nearly 70%) in replica set cluster. All nodes running seperately on m1.medium in availbility zone us-east-1a. All nodes using instance storage Max wait time increased to 240s from default 120s Remote standalone mongodb instance. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 40 885 60 0 30.9 MB NORMAL true true 1 32 59 787 60 0 27.5 MB JOURNAL_SAFE true true 1 32 49 788 60 0 27.5 MB FSYNC_SAFE true true 1 32 32 848 60 0 29.6 MB SAFE true true 1 32 61 845 60 1 29.5 MB NONE Three member replica set with primary + 2 secondary. Each node is running on seperate ec2 instance. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 1 324 60 0 11.3 MB NORMAL true true 1 32 12 229 60 0 8.0 MB MAJORITY true true 1 32 14 268 60 0 9.4 MB JOURNAL_SAFE true true 1 32 13 262 60 0 9.2 MB FSYNC_SAFE true true 1 32 9 273 60 0 9.5 MB SAFE true true 1 32 3 328 60 0 11.5 MB NONE true true 1 32 9 228 60 0 8.0 MB REPLICAS_SAFE
          Hide
          Shashank Gupta added a comment - - edited

          Write throughput comparision between standalone vs replica set on increasing write threads. Throughput decreases from 33% to 75%.

          • Remote standalone mongodb instance.
            remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
            true true 1 4 508 31 5 101 13.0 MB SAFE
            true true 1 8 290 32 5 58 13.4 MB SAFE
            true true 1 16 5 39 5 1 16.4 MB SAFE
            true true 1 32 4 54 5 0 22.6 MB SAFE
          • Three member replica set with primary| secondary and arbiter running on single remote machine.
            remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
            true true 1 4 70 7 5 14 2.9 MB SAFE
            true true 1 8 80 13 5 16 5.5 MB SAFE
            true true 1 16 4 20 5 0 8.4 MB SAFE
            true true 1 32 5 38 5 1 15.9 MB SAFE
          Show
          Shashank Gupta added a comment - - edited Write throughput comparision between standalone vs replica set on increasing write threads. Throughput decreases from 33% to 75%. Remote standalone mongodb instance. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 4 508 31 5 101 13.0 MB SAFE true true 1 8 290 32 5 58 13.4 MB SAFE true true 1 16 5 39 5 1 16.4 MB SAFE true true 1 32 4 54 5 0 22.6 MB SAFE Three member replica set with primary| secondary and arbiter running on single remote machine. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 4 70 7 5 14 2.9 MB SAFE true true 1 8 80 13 5 16 5.5 MB SAFE true true 1 16 4 20 5 0 8.4 MB SAFE true true 1 32 5 38 5 1 15.9 MB SAFE
          Hide
          Shashank Gupta added a comment - - edited

          Write throughput decreased by approximate 33% from standalone mongodb to mongodb replica set.

          • Remote standalone mongodb instance.
            remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
            true true 1 32 5 53 5 1 22.2 MB NORMAL
            true true 1 32 5 54 5 1 22.6 MB JOURNAL_SAFE
            true true 1 32 4 53 5 0 22.2 MB FSYNC_SAFE
            true true 1 32 3 54 5 0 22.6 MB SAFE
            true true 1 32 2 52 5 0 21.8 MB NONE
          • Three member replica set with primary, secondary and arbiter running on single remote machine.
            remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
            true true 1 32 2 48 5 0 20.1 MB NORMAL
            true true 1 32 4 32 5 0 13.4 MB MAJORITY
            true true 1 32 4 34 5 0 14.3 MB JOURNAL_SAFE
            true true 1 32 4 35 5 0 14.7 MB FSYNC_SAFE
            true true 1 32 4 36 5 0 15.1 MB SAFE
            true true 1 32 4 33 5 0 13.8 MB REPLICAS_SAFE
            true true 1 32 5 48 5 1 20.1 MB NONE
          Show
          Shashank Gupta added a comment - - edited Write throughput decreased by approximate 33% from standalone mongodb to mongodb replica set. Remote standalone mongodb instance. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 5 53 5 1 22.2 MB NORMAL true true 1 32 5 54 5 1 22.6 MB JOURNAL_SAFE true true 1 32 4 53 5 0 22.2 MB FSYNC_SAFE true true 1 32 3 54 5 0 22.6 MB SAFE true true 1 32 2 52 5 0 21.8 MB NONE Three member replica set with primary, secondary and arbiter running on single remote machine. remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 1 32 2 48 5 0 20.1 MB NORMAL true true 1 32 4 32 5 0 13.4 MB MAJORITY true true 1 32 4 34 5 0 14.3 MB JOURNAL_SAFE true true 1 32 4 35 5 0 14.7 MB FSYNC_SAFE true true 1 32 4 36 5 0 15.1 MB SAFE true true 1 32 4 33 5 0 13.8 MB REPLICAS_SAFE true true 1 32 5 48 5 1 20.1 MB NONE
          Hide
          Chetan Mehrotra added a comment -

          It might be interesting to know the performance with different write concerns (just this dimension).

          Update the test in rev 1539600. Below are results for 0 reader, 2 writer, 4 MB per insert for various write concern against a remote Mongo server

          remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern
          true true 0 2 0 16 5 0 13.4 MB NORMAL
          true true 0 2 0 14 5 0 11.7 MB JOURNAL_SAFE
          true true 0 2 0 14 5 0 11.7 MB FSYNC_SAFE
          true true 0 2 0 15 5 0 12.6 MB SAFE
          true true 0 2 0 16 5 0 13.4 MB NONE
          Show
          Chetan Mehrotra added a comment - It might be interesting to know the performance with different write concerns (just this dimension). Update the test in rev 1539600 . Below are results for 0 reader, 2 writer, 4 MB per insert for various write concern against a remote Mongo server remote samedb readers writers reads writes time readThroughPut writeThroughPut writeConcern true true 0 2 0 16 5 0 13.4 MB NORMAL true true 0 2 0 14 5 0 11.7 MB JOURNAL_SAFE true true 0 2 0 14 5 0 11.7 MB FSYNC_SAFE true true 0 2 0 15 5 0 12.6 MB SAFE true true 0 2 0 16 5 0 13.4 MB NONE
          Hide
          Thomas Mueller added a comment -

          This is very interesting! Thanks a lot.

          It might be interesting to know the performance with different write concerns (just this dimension).

          > if writing blob in separate collection provides any benefit

          I think you mean "separate database"?

          Show
          Thomas Mueller added a comment - This is very interesting! Thanks a lot. It might be interesting to know the performance with different write concerns (just this dimension). > if writing blob in separate collection provides any benefit I think you mean "separate database"?
          Hide
          Chetan Mehrotra added a comment -

          Added initial test in http://svn.apache.org/r1539551

          Running the test yields following result

          remote samedb readers writers totalReads totalWrites time readThroughPut writeThroughPut
          false false 5 0 280796 0 5 56159 0 B
          false false 5 1 13312 56 5 2662 23.5 MB
          false false 5 2 3922 115 5 784 48.2 MB
          false false 5 4 1453 136 5 290 57.0 MB
          false false 10 0 392799 0 5 78559 0 B
          false false 10 1 37530 153 5 7506 64.2 MB
          false false 10 2 19955 171 5 3991 71.7 MB
          false false 10 4 13019 175 5 2603 73.4 MB
          false false 15 0 334740 0 5 66948 0 B
          false false 15 1 72733 180 5 14546 75.5 MB
          false false 15 2 40950 129 5 8190 54.1 MB
          false false 15 4 17759 189 5 3551 79.3 MB
          false false 20 0 344517 0 5 68903 0 B
          false false 20 1 93679 188 5 18735 78.9 MB
          false false 20 2 60633 192 5 12126 80.5 MB
          false false 20 4 45535 199 5 9107 83.5 MB
          false true 5 0 367334 0 5 73466 0 B
          false true 5 1 27254 36 5 5450 15.1 MB
          false true 5 2 2633 101 5 526 42.4 MB
          false true 5 4 1023 146 5 204 61.2 MB
          false true 10 0 361794 0 5 72358 0 B
          false true 10 1 39355 164 5 7871 68.8 MB
          false true 10 2 22363 179 5 4472 75.1 MB
          false true 10 4 11117 183 5 2223 76.8 MB
          false true 15 0 371182 0 5 74236 0 B
          false true 15 1 59039 164 5 11807 68.8 MB
          false true 15 2 42343 189 5 8468 79.3 MB
          false true 15 4 28134 151 5 5626 63.3 MB
          false true 20 0 353480 0 5 70696 0 B
          false true 20 1 79008 191 5 15801 80.1 MB
          false true 20 2 49861 124 5 9972 52.0 MB
          false true 20 4 30017 198 5 6003 83.0 MB
          true false 5 0 31324 0 5 6264 0 B
          true false 5 1 2763 28 5 552 11.7 MB
          true false 5 2 1313 29 5 262 12.2 MB
          true false 5 4 714 31 5 142 13.0 MB
          true false 10 0 73676 0 5 14735 0 B
          true false 10 1 5618 28 5 1123 11.7 MB
          true false 10 2 5504 28 5 1100 11.7 MB
          true false 10 4 911 30 5 182 12.6 MB
          true false 15 0 72758 0 5 14551 0 B
          true false 15 1 5644 27 5 1128 11.3 MB
          true false 15 2 3427 28 5 685 11.7 MB
          true false 15 4 1104 30 5 220 12.6 MB
          true false 20 0 74050 0 5 14810 0 B
          true false 20 1 12430 24 5 2486 10.1 MB
          true false 20 2 2888 29 5 577 12.2 MB
          true false 20 4 1253 30 5 250 12.6 MB
          true true 5 0 31143 0 5 6228 0 B
          true true 5 1 2613 28 5 522 11.7 MB
          true true 5 2 1213 29 5 242 12.2 MB
          true true 5 4 2768 26 5 553 10.9 MB
          true true 10 0 73625 0 5 14725 0 B
          true true 10 1 16300 23 5 3260 9.6 MB
          true true 10 2 2221 29 5 444 12.2 MB
          true true 10 4 865 30 5 173 12.6 MB
          true true 15 0 73177 0 5 14635 0 B
          true true 15 1 5214 27 5 1042 11.3 MB
          true true 15 2 6023 27 5 1204 11.3 MB
          true true 15 4 1113 31 5 222 13.0 MB
          true true 20 0 73911 0 5 14782 0 B
          true true 20 1 5984 27 5 1196 11.3 MB
          true true 20 2 2914 29 5 582 12.2 MB
          true true 20 4 1353 31 5 270 13.0 MB
          Show
          Chetan Mehrotra added a comment - Added initial test in http://svn.apache.org/r1539551 Running the test yields following result remote samedb readers writers totalReads totalWrites time readThroughPut writeThroughPut false false 5 0 280796 0 5 56159 0 B false false 5 1 13312 56 5 2662 23.5 MB false false 5 2 3922 115 5 784 48.2 MB false false 5 4 1453 136 5 290 57.0 MB false false 10 0 392799 0 5 78559 0 B false false 10 1 37530 153 5 7506 64.2 MB false false 10 2 19955 171 5 3991 71.7 MB false false 10 4 13019 175 5 2603 73.4 MB false false 15 0 334740 0 5 66948 0 B false false 15 1 72733 180 5 14546 75.5 MB false false 15 2 40950 129 5 8190 54.1 MB false false 15 4 17759 189 5 3551 79.3 MB false false 20 0 344517 0 5 68903 0 B false false 20 1 93679 188 5 18735 78.9 MB false false 20 2 60633 192 5 12126 80.5 MB false false 20 4 45535 199 5 9107 83.5 MB false true 5 0 367334 0 5 73466 0 B false true 5 1 27254 36 5 5450 15.1 MB false true 5 2 2633 101 5 526 42.4 MB false true 5 4 1023 146 5 204 61.2 MB false true 10 0 361794 0 5 72358 0 B false true 10 1 39355 164 5 7871 68.8 MB false true 10 2 22363 179 5 4472 75.1 MB false true 10 4 11117 183 5 2223 76.8 MB false true 15 0 371182 0 5 74236 0 B false true 15 1 59039 164 5 11807 68.8 MB false true 15 2 42343 189 5 8468 79.3 MB false true 15 4 28134 151 5 5626 63.3 MB false true 20 0 353480 0 5 70696 0 B false true 20 1 79008 191 5 15801 80.1 MB false true 20 2 49861 124 5 9972 52.0 MB false true 20 4 30017 198 5 6003 83.0 MB true false 5 0 31324 0 5 6264 0 B true false 5 1 2763 28 5 552 11.7 MB true false 5 2 1313 29 5 262 12.2 MB true false 5 4 714 31 5 142 13.0 MB true false 10 0 73676 0 5 14735 0 B true false 10 1 5618 28 5 1123 11.7 MB true false 10 2 5504 28 5 1100 11.7 MB true false 10 4 911 30 5 182 12.6 MB true false 15 0 72758 0 5 14551 0 B true false 15 1 5644 27 5 1128 11.3 MB true false 15 2 3427 28 5 685 11.7 MB true false 15 4 1104 30 5 220 12.6 MB true false 20 0 74050 0 5 14810 0 B true false 20 1 12430 24 5 2486 10.1 MB true false 20 2 2888 29 5 577 12.2 MB true false 20 4 1253 30 5 250 12.6 MB true true 5 0 31143 0 5 6228 0 B true true 5 1 2613 28 5 522 11.7 MB true true 5 2 1213 29 5 242 12.2 MB true true 5 4 2768 26 5 553 10.9 MB true true 10 0 73625 0 5 14725 0 B true true 10 1 16300 23 5 3260 9.6 MB true true 10 2 2221 29 5 444 12.2 MB true true 10 4 865 30 5 173 12.6 MB true true 15 0 73177 0 5 14635 0 B true true 15 1 5214 27 5 1042 11.3 MB true true 15 2 6023 27 5 1204 11.3 MB true true 15 4 1113 31 5 222 13.0 MB true true 20 0 73911 0 5 14782 0 B true true 20 1 5984 27 5 1196 11.3 MB true true 20 2 2914 29 5 582 12.2 MB true true 20 4 1353 31 5 270 13.0 MB

            People

            • Assignee:
              Chetan Mehrotra
              Reporter:
              Chetan Mehrotra
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development