Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-6553

Benchmark counter improvements (counters++)

    Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 2.1 beta2
    • Component/s: None
    • Labels:

      Description

      Benchmark the difference in performance between CASSANDRA-6504 and trunk.

      • Updating totally unrelated counters (different partitions)
      • Updating the same counters a lot (same cells in the same partition)
      • Different cells in the same few partitions (hot counter partition)

      benchmark: https://github.com/apache/cassandra/tree/1218bcacba7edefaf56cf8440d0aea5794c89a1e (old counters)
      compared to: https://github.com/apache/cassandra/tree/714c423360c36da2a2b365efaf9c5c4f623ed133 (new counters)

      So far, the above changes should only affect the write path.

      1. 6553.txt
        2 kB
        Benedict
      2. 6553.uber.quorum.bdplab.read.png
        73 kB
        Ryan McGuire
      3. 6553.uber.quorum.bdplab.write.png
        82 kB
        Ryan McGuire
      4. high_cl_one.png
        73 kB
        Russ Hatch
      5. high_cl_quorum.png
        75 kB
        Russ Hatch
      6. logs.tar.gz
        3.38 MB
        Russ Hatch
      7. low_cl_one.png
        81 kB
        Russ Hatch
      8. low_cl_quorum.png
        81 kB
        Russ Hatch
      9. tracing.txt
        8 kB
        Russ Hatch
      10. uber_cl_one.png
        78 kB
        Russ Hatch
      11. uber_cl_quorum.png
        79 kB
        Russ Hatch

        Issue Links

          Activity

          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          To clarify - we expect that (under contention) the latency will go up, and the throughput will decrease. We need to quantify the change, however.

          Show
          iamaleksey Aleksey Yeschenko added a comment - To clarify - we expect that (under contention) the latency will go up, and the throughput will decrease. We need to quantify the change, however.
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          Also, need to see the results at RF>1 (RF=1 is not interesting), and separately for CL.ONE and CL.QUORUM.

          Show
          iamaleksey Aleksey Yeschenko added a comment - Also, need to see the results at RF>1 (RF=1 is not interesting), and separately for CL.ONE and CL.QUORUM.
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          Or, really, just 2.0 vs. 2.1, since that's what really matters.

          Show
          iamaleksey Aleksey Yeschenko added a comment - Or, really, just 2.0 vs. 2.1, since that's what really matters.
          Hide
          iamaleksey Aleksey Yeschenko added a comment - - edited

          Russ Hatch Is there any progress on this? Can you benchmark cassandra-2.0 branch vs. https://github.com/iamaleksey/cassandra/commits/6506-nocompat ? It's becoming kinda pressing :\ Thanks.

          Show
          iamaleksey Aleksey Yeschenko added a comment - - edited Russ Hatch Is there any progress on this? Can you benchmark cassandra-2.0 branch vs. https://github.com/iamaleksey/cassandra/commits/6506-nocompat ? It's becoming kinda pressing :\ Thanks.
          Hide
          rhatch Russ Hatch added a comment -

          Aleksey Yeschenko Sorry for the delay. This is a high priority for me this week, so I hope to have some more progress soon. Took me a while to figure out how to accomplish this with the new stress tool. Below I've included the stress comparisons I plan to do.

          Counter write performance
          
              rf>1 and cl.one; rf>1 and cl.quorum
          
              contention incrementing counters in different partitions:
                  # two counters, each with a different partition
                  ./bin/cassandra-stress counterwrite n=10000 -key populate=1..2 -col n=FIXED\(1\)
          
              contention incrementing same counter (same partition and cell):
                  # contention over one counter
                  ./bin/cassandra-stress counterwrite n=1000000 -key dist=FIXED\(1\) -col n=FIXED\(1\)
          
              contention incrementing one partition with two cells:
                  # two counters in same partition
                  ./bin/cassandra-stress counterwrite n=1000000 -key dist=FIXED\(1\) -col n=FIXED\(1\) -col n=FIXED\(2\)
          
              contention incrementing counters in a few cells in the same few partitions (hot counter partition):
                  # three partitions with three cells (9 counters)
                  ./bin/cassandra-stress counterwrite n=1000000 -key populate=1..3 -col n=FIXED\(1\) -col n=FIXED\(2\) -col n=FIXED\(3\)
          
          Show
          rhatch Russ Hatch added a comment - Aleksey Yeschenko Sorry for the delay. This is a high priority for me this week, so I hope to have some more progress soon. Took me a while to figure out how to accomplish this with the new stress tool. Below I've included the stress comparisons I plan to do. Counter write performance rf>1 and cl.one; rf>1 and cl.quorum contention incrementing counters in different partitions: # two counters, each with a different partition ./bin/cassandra-stress counterwrite n=10000 -key populate=1..2 -col n=FIXED\(1\) contention incrementing same counter (same partition and cell): # contention over one counter ./bin/cassandra-stress counterwrite n=1000000 -key dist=FIXED\(1\) -col n=FIXED\(1\) contention incrementing one partition with two cells: # two counters in same partition ./bin/cassandra-stress counterwrite n=1000000 -key dist=FIXED\(1\) -col n=FIXED\(1\) -col n=FIXED\(2\) contention incrementing counters in a few cells in the same few partitions (hot counter partition): # three partitions with three cells (9 counters) ./bin/cassandra-stress counterwrite n=1000000 -key populate=1..3 -col n=FIXED\(1\) -col n=FIXED\(2\) -col n=FIXED\(3\)
          Hide
          benedict Benedict added a comment -

          ./bin/cassandra-stress counterwrite n=1000000 -key populate=1..3 -col n=FIXED(1) -col n=FIXED(2) -col n=FIXED(3)

          This should throw an error. If it doesn't, it's a bug. Defining col three times doesn't mean you get three different col definitions. Unless this is a typo, you'd want something like:

          -col n=uniform(1..3)

          Although perhaps we should introduce a new distribution that walks through all values, as populate does for -key.

          Also, your first two tests look particularly similar; it's unlikely one will yield any more useful information than another (N threads to two partitions is probably roughly the same as N/2 threads to one, unless we saturate the network)

          It might be worth throwing a mixed workload in there, to shake things up a little, e.g. (50/50 split):

          cassandra-stress mixed clustering=exp(1..10) ratio(counterread=1,counterwrite=1)

          Looking at this, I realise the "clustering" option is poorly documented, in fact it doesn't say anything about what it does. I'll fix that. It "clusters" commands; i.e. when selecting a new command, defines how many of that command type will be executed before a different command type is selected.

          Show
          benedict Benedict added a comment - ./bin/cassandra-stress counterwrite n=1000000 -key populate=1..3 -col n=FIXED(1) -col n=FIXED(2) -col n=FIXED(3) This should throw an error. If it doesn't, it's a bug. Defining col three times doesn't mean you get three different col definitions. Unless this is a typo, you'd want something like: -col n=uniform(1..3) Although perhaps we should introduce a new distribution that walks through all values, as populate does for -key. Also, your first two tests look particularly similar; it's unlikely one will yield any more useful information than another (N threads to two partitions is probably roughly the same as N/2 threads to one, unless we saturate the network) It might be worth throwing a mixed workload in there, to shake things up a little, e.g. (50/50 split): cassandra-stress mixed clustering=exp(1..10) ratio(counterread=1,counterwrite=1) Looking at this, I realise the "clustering" option is poorly documented, in fact it doesn't say anything about what it does. I'll fix that. It "clusters" commands; i.e. when selecting a new command, defines how many of that command type will be executed before a different command type is selected.
          Hide
          rhatch Russ Hatch added a comment -

          Those four commands all run without error. If I try the uniform syntax suggested above an exception bubbles up:

          java.lang.IllegalStateException: Variable column counts not supported for counters
          	at org.apache.cassandra.stress.operations.ThriftCounterAdder.<init>(ThriftCounterAdder.java:37)
          	at org.apache.cassandra.stress.StressAction.createOperation(StressAction.java:493)
          	at org.apache.cassandra.stress.StressAction.createOperation(StressAction.java:446)
          	at org.apache.cassandra.stress.StressAction.access$300(StressAction.java:38)
          	at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:292)
          

          I think that's how I settled on the funky syntax you noticed above. I tried many different combinations but couldn't seem to get the partition/cell counts I wanted any other way.

          Show
          rhatch Russ Hatch added a comment - Those four commands all run without error. If I try the uniform syntax suggested above an exception bubbles up: java.lang.IllegalStateException: Variable column counts not supported for counters at org.apache.cassandra.stress.operations.ThriftCounterAdder.<init>(ThriftCounterAdder.java:37) at org.apache.cassandra.stress.StressAction.createOperation(StressAction.java:493) at org.apache.cassandra.stress.StressAction.createOperation(StressAction.java:446) at org.apache.cassandra.stress.StressAction.access$300(StressAction.java:38) at org.apache.cassandra.stress.StressAction$Consumer.run(StressAction.java:292) I think that's how I settled on the funky syntax you noticed above. I tried many different combinations but couldn't seem to get the partition/cell counts I wanted any other way.
          Hide
          benedict Benedict added a comment -

          Those four commands all run without error.

          It'll be selecting the last -col definition you provide then (so in the case I pasted, it'll be same as -coln=FIXED(3)). I'll file a ticket and post a fix tomorrow for not erroring out.

          I'll also have a look and see if we can't make the number of columns variable for counters. It does make sense to write to a random selection of the possible counters for a single partition key, and it may have just been laziness in not alleviating a prior limitation when I updated everything.

          Show
          benedict Benedict added a comment - Those four commands all run without error. It'll be selecting the last -col definition you provide then (so in the case I pasted, it'll be same as -coln=FIXED(3)). I'll file a ticket and post a fix tomorrow for not erroring out. I'll also have a look and see if we can't make the number of columns variable for counters. It does make sense to write to a random selection of the possible counters for a single partition key, and it may have just been laziness in not alleviating a prior limitation when I updated everything.
          Hide
          benedict Benedict added a comment -

          FTR, if you want to run a mixed or read workload with counters, make sure you apply CASSANDRA-6835

          Show
          benedict Benedict added a comment - FTR, if you want to run a mixed or read workload with counters, make sure you apply CASSANDRA-6835
          Hide
          rhatch Russ Hatch added a comment - - edited

          Show
          rhatch Russ Hatch added a comment - - edited
          Hide
          rhatch Russ Hatch added a comment -

          These tests were done with a 4 node cluster using RF=3. We picked 3 scenarios (low, high, and uber contention) and I ran each with CL=one and CL=quorum, with counterwrite and counterread.

          New counters appeared to outperform 2.0 in every aspect measured, except in the case of competing counter reads in the same partition (aka 'uber contention').

          All tests were done with 50 threads.

          For "low contention" I tested incrementing/reading 10k counters all in separate partitions (50 threads)
          For "high contention" I tested incrementing/reading a single counter cell (50 threads)
          For "uber contention" I tested incrementing/reading 100 counter cells within the same partition (50 threads). I first tried this scenario with 10k cells in the same partition but performance was degraded so badly I don't think the data would be useful, so I bumped the cells clear down to 100.

          Interactive Graphs:

          CL=one:
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_low_cl_one.json&metric=op_rate&operation=counterwrite&smoothing=1
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_high_cl_one.json&metric=op_rate&operation=counterwrite&smoothing=1
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_uber_cl_one.json&metric=op_rate&operation=counterwrite&smoothing=1

          CL=quorum:
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_low_cl_quorum.json&metric=op_rate&operation=counterwrite&smoothing=1
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_high_cl_quorum.json&metric=op_rate&operation=counterwrite&smoothing=1
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_uber_cl_quorum.json&metric=op_rate&operation=counterwrite&smoothing=1

          Show
          rhatch Russ Hatch added a comment - These tests were done with a 4 node cluster using RF=3. We picked 3 scenarios (low, high, and uber contention) and I ran each with CL=one and CL=quorum, with counterwrite and counterread. New counters appeared to outperform 2.0 in every aspect measured, except in the case of competing counter reads in the same partition (aka 'uber contention'). All tests were done with 50 threads. For "low contention" I tested incrementing/reading 10k counters all in separate partitions (50 threads) For "high contention" I tested incrementing/reading a single counter cell (50 threads) For "uber contention" I tested incrementing/reading 100 counter cells within the same partition (50 threads). I first tried this scenario with 10k cells in the same partition but performance was degraded so badly I don't think the data would be useful, so I bumped the cells clear down to 100. Interactive Graphs: CL=one: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_low_cl_one.json&metric=op_rate&operation=counterwrite&smoothing=1 http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_high_cl_one.json&metric=op_rate&operation=counterwrite&smoothing=1 http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_uber_cl_one.json&metric=op_rate&operation=counterwrite&smoothing=1 CL=quorum: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_low_cl_quorum.json&metric=op_rate&operation=counterwrite&smoothing=1 http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_high_cl_quorum.json&metric=op_rate&operation=counterwrite&smoothing=1 http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_uber_cl_quorum.json&metric=op_rate&operation=counterwrite&smoothing=1
          Hide
          jbellis Jonathan Ellis added a comment -

          New counters appeared to outperform 2.0 in every aspect measured, except in the case of competing counter reads in the same partition (aka 'uber contention').

          Am I misreading something? That sure looks to me like 2.1 is almost twice as fast as 2.0.

          I'm especially pleased at how even 2.1 performance is instead of the large variance in 2.0.

          Show
          jbellis Jonathan Ellis added a comment - New counters appeared to outperform 2.0 in every aspect measured, except in the case of competing counter reads in the same partition (aka 'uber contention'). Am I misreading something? That sure looks to me like 2.1 is almost twice as fast as 2.0. I'm especially pleased at how even 2.1 performance is instead of the large variance in 2.0.
          Hide
          rhatch Russ Hatch added a comment -

          New counters appeared to outperform 2.0 in every aspect measured, except in the case of competing counter reads in the same partition (aka 'uber contention').

          That is specifically for reads with high contention. Writes did outperform 2.0 (writes are show in the image above)

          Show
          rhatch Russ Hatch added a comment - New counters appeared to outperform 2.0 in every aspect measured, except in the case of competing counter reads in the same partition (aka 'uber contention'). That is specifically for reads with high contention. Writes did outperform 2.0 (writes are show in the image above)
          Hide
          enigmacurry Ryan McGuire added a comment -

          I'm seeing similar results to Russ Hatch on different hardware (bdplab):

          data is here

          Show
          enigmacurry Ryan McGuire added a comment - I'm seeing similar results to Russ Hatch on different hardware (bdplab): data is here
          Hide
          slebresne Sylvain Lebresne added a comment -

          The reads results are somewhat weird and warrant some further investigation imo: the current change shouldn't have modified the read path if I'm not mistaken and so we shouldn't see much (or any really) difference there.

          On the write side, the results are definitively nice to see, but I'd be curious to see some comparison of with and without the (new) counter cache, just to get an idea of what happen when things don't fit in said cache anymore.

          Show
          slebresne Sylvain Lebresne added a comment - The reads results are somewhat weird and warrant some further investigation imo: the current change shouldn't have modified the read path if I'm not mistaken and so we shouldn't see much (or any really) difference there. On the write side, the results are definitively nice to see, but I'd be curious to see some comparison of with and without the (new) counter cache, just to get an idea of what happen when things don't fit in said cache anymore.
          Hide
          rhatch Russ Hatch added a comment -

          So it seems like the wishlist for more testing might include:

          • mixed read/write test(s) (maybe a write-heavy and a read-heavy profile, say 10:1)
          • Aleksey Yeschenko mentioned "there are some other scenarios to test - when the read before write hits the disk, for example" (not sure how to approach testing this – or would this be exercised by a read-heavy workload mentioned above?)
          • tests with counter cache disabled, mentioned by Sylvain Lebresne (would this require a patched version of cassandra or is there a simpler way to do this?)

          any comments on the above or other scenarios I can help out with?

          Show
          rhatch Russ Hatch added a comment - So it seems like the wishlist for more testing might include: mixed read/write test(s) (maybe a write-heavy and a read-heavy profile, say 10:1) Aleksey Yeschenko mentioned "there are some other scenarios to test - when the read before write hits the disk, for example" (not sure how to approach testing this – or would this be exercised by a read-heavy workload mentioned above?) tests with counter cache disabled, mentioned by Sylvain Lebresne (would this require a patched version of cassandra or is there a simpler way to do this?) any comments on the above or other scenarios I can help out with?
          Hide
          rhatch Russ Hatch added a comment -

          I investigated a bit more for the uber-contention reads (locally using aleksey's counters branch), using settraceprobability 0.001.

          I'm not sure what the getSlice call should look like, but I attached the output I got from the session and events if they are of any use.

          Show
          rhatch Russ Hatch added a comment - I investigated a bit more for the uber-contention reads (locally using aleksey's counters branch), using settraceprobability 0.001. I'm not sure what the getSlice call should look like, but I attached the output I got from the session and events if they are of any use.
          Hide
          slebresne Sylvain Lebresne added a comment -

          there are some other scenarios to test - when the read before write hits the disk, for example

          That's actually largely what I meant by testing with the counter cache disabled. The new counter implem has a (new) counter cache that saves the read-before-write from hitting disk on a cache hit. So a simple way to check what happen when you run out of cache would be to just disable it, which can be done from the yaml (check the counter_cache_* new options).

          Show
          slebresne Sylvain Lebresne added a comment - there are some other scenarios to test - when the read before write hits the disk, for example That's actually largely what I meant by testing with the counter cache disabled. The new counter implem has a (new) counter cache that saves the read-before-write from hitting disk on a cache hit. So a simple way to check what happen when you run out of cache would be to just disable it, which can be done from the yaml (check the counter_cache_* new options).
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          Sylvain Lebresne Actually, no. I meant going further. Obviously disable the cache, but also make sure that we are reading from disk, not the memtable. Maybe go even further and test with data not fitting the page cache, too.

          Show
          iamaleksey Aleksey Yeschenko added a comment - Sylvain Lebresne Actually, no. I meant going further. Obviously disable the cache, but also make sure that we are reading from disk, not the memtable. Maybe go even further and test with data not fitting the page cache, too.
          Show
          rhatch Russ Hatch added a comment - Here's new graphs for the same 6 scenarios as above, but with counter_cache_size_in_mb set to zero to disable caching: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_high_cl_one_no_counter_cache.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_high_cl_quorum_no_counter_cache.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_low_cl_one_no_counter_cache.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_low_cl_quorum_no_counter_cache.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_uber_cl_one_no_counter_cache.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_uber_cl_quorum_no_counter_cache.json
          Show
          rhatch Russ Hatch added a comment - - edited graphs comparing Aleksey Yeschenko 's 2.1 branch against the 2.0 branch (with counter caching disabled): http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_high_cl_one_no_counter_cache_20vs21.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_high_cl_quorum_no_counter_cache_20vs21.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_low_cl_one_no_counter_cache_20vs21.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_low_cl_quorum_no_counter_cache_20vs21.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_uber_cl_one_no_counter_cache_20vs21.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.counter_uber_cl_quorum_no_counter_cache_20vs21.json
          Hide
          rhatch Russ Hatch added a comment -

          more test data, comparing 2.0 vs 2.1, with count cache disabled:

          The "uber contention" scenario was corrected for these tests. Earlier tests were updating all 100 contended cells in each operation. These tests now update one cell per operation, chosen from a uniform distribution of the 100 cells. This was the original intention of the uber contention tests, but I was using cassandra-stress incorrectly earlier.

          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_one.json
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_one.json
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.uber_contention_no_cache_CL_one.json

          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_quorum.json
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_quorum.json
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_no_cache_CL_quorum.json

          uber contention counter read and writes both show some odd results. CL.ONE writes degraded until stress gave up on the test.

          Show
          rhatch Russ Hatch added a comment - more test data, comparing 2.0 vs 2.1, with count cache disabled: The "uber contention" scenario was corrected for these tests. Earlier tests were updating all 100 contended cells in each operation. These tests now update one cell per operation, chosen from a uniform distribution of the 100 cells. This was the original intention of the uber contention tests, but I was using cassandra-stress incorrectly earlier. http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_one.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_one.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.uber_contention_no_cache_CL_one.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_quorum.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_quorum.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_no_cache_CL_quorum.json uber contention counter read and writes both show some odd results. CL.ONE writes degraded until stress gave up on the test.
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          Russ Hatch Thanks. Counter writes at uber contention will look decent (in these graphs) with CASSANDRA-6880, but why the slower reads - I have no idea. The read path for counters hasn't really changed since 2.0, so ¯_(ツ)_/¯.

          Could be CASSANDRA-6746, could be something else. As I said before, I'd like to see the same a similar set of benchmarks on non-counter tables, to see if regular reads would show same weird read results 'under contention'.

          Show
          iamaleksey Aleksey Yeschenko added a comment - Russ Hatch Thanks. Counter writes at uber contention will look decent (in these graphs) with CASSANDRA-6880 , but why the slower reads - I have no idea. The read path for counters hasn't really changed since 2.0, so ¯_(ツ)_/¯. Could be CASSANDRA-6746 , could be something else. As I said before, I'd like to see the same a similar set of benchmarks on non-counter tables, to see if regular reads would show same weird read results 'under contention'.
          Hide
          rhatch Russ Hatch added a comment -

          Aleksey Yeschenko – Thought you might want to know I repeated the test w/the degraded performance and that problem did not repeat this time around.

          I should note these tests are still using stress w/thrift (can't get native cql3 to work with stress yet).

          I'll attempt to run some regular read/write performance tests now for comparison, so we can see if that's where the bottleneck might be.

          Show
          rhatch Russ Hatch added a comment - Aleksey Yeschenko – Thought you might want to know I repeated the test w/the degraded performance and that problem did not repeat this time around. I should note these tests are still using stress w/thrift (can't get native cql3 to work with stress yet). I'll attempt to run some regular read/write performance tests now for comparison, so we can see if that's where the bottleneck might be.
          Show
          rhatch Russ Hatch added a comment - - edited For comparison, here's graphs of 2.0 vs 2.1 for regular write/read operations ( not counterread or counterwrite). http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.read_write.low_contention_CL_one.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.read_write.high_contention_CL_one.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.read_write.uber_contention_CL_one.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.read_write.low_contention_CL_quorum.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.read_write.high_contention_CL_quorum.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.read_write.uber_contention_CL_quorum.json I forgot to mention that in the uber contention scenarios, I had to drop the operation count down to 1.4m because around 1.5m operations there's a decline in performance and the test couldn't complete and didn't seem to recover.
          Show
          iamaleksey Aleksey Yeschenko added a comment - Benedict http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.read_write.uber_contention_CL_one.json&metric=op_rate&operation=read&smoothing=1&xmin=0&xmax=91.85&ymin=0&ymax=35426.6 these are non-counter reads. Is stress whack, or do you have any ideas on what's causing the regression?
          Hide
          benedict Benedict added a comment -

          AbstractCType.compare() seems to be eating up a lot of time. The attached patch optimises SimpleSparseCellNameType, which on my box brings performance in line with 2.0 for this workload. I haven't attempted to improve performance across the board for AbstractCType; it probably requires a separate ticket to address performance improvements.

          Show
          benedict Benedict added a comment - AbstractCType.compare() seems to be eating up a lot of time. The attached patch optimises SimpleSparseCellNameType, which on my box brings performance in line with 2.0 for this workload . I haven't attempted to improve performance across the board for AbstractCType; it probably requires a separate ticket to address performance improvements.
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          Russ Hatch I want to revisit the stress params, later, but for now, could we compare these 3 branches against each other?

          1. Stock 2.0 (https://github.com/apache/cassandra/tree/cassandra-2.0)
          2. Stock 2.1 + Benedict's patch (https://github.com/iamaleksey/cassandra/tree/6553)
          3. 2.1 with CASSANDRA-6880 and Benedict's patch (https://github.com/iamaleksey/cassandra/tree/6880)

          Two runs - one w/out the counter cache, and another one with counter cache on.

          In the meantime I'll think about the new stress params I'd like to run the benchmarks with.

          Thanks!

          Show
          iamaleksey Aleksey Yeschenko added a comment - Russ Hatch I want to revisit the stress params, later, but for now, could we compare these 3 branches against each other? 1. Stock 2.0 ( https://github.com/apache/cassandra/tree/cassandra-2.0 ) 2. Stock 2.1 + Benedict's patch ( https://github.com/iamaleksey/cassandra/tree/6553 ) 3. 2.1 with CASSANDRA-6880 and Benedict's patch ( https://github.com/iamaleksey/cassandra/tree/6880 ) Two runs - one w/out the counter cache, and another one with counter cache on. In the meantime I'll think about the new stress params I'd like to run the benchmarks with. Thanks!
          Hide
          benedict Benedict added a comment -

          Would be good to throw in at least one new test that compares large number of cell updates across a wide range of partitions (so ostensibly uncontended). 6880 should make this slower, but would be good to quantify it

          Show
          benedict Benedict added a comment - Would be good to throw in at least one new test that compares large number of cell updates across a wide range of partitions (so ostensibly uncontended). 6880 should make this slower, but would be good to quantify it
          Show
          rhatch Russ Hatch added a comment - No counter cache: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.uber_contention_no_cache_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_quorum_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_quorum_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_no_cache_CL_quorum_3_28.json Counter cache enabled: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.uber_contention_CL_one_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_CL_quorum_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_CL_quorum_3_28.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_CL_quorum_3_28.json
          Hide
          rhatch Russ Hatch added a comment -

          I think some of the write scenarios were running into CASSANDRA-6943.

          Show
          rhatch Russ Hatch added a comment - I think some of the write scenarios were running into CASSANDRA-6943 .
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          I think some of the write scenarios were running into CASSANDRA-6943.

          If it were only 6800 stalling, I'd assume I just messed up with lock grabbing order, but with both 2.1 branches failing, I think you are correct.

          Show
          iamaleksey Aleksey Yeschenko added a comment - I think some of the write scenarios were running into CASSANDRA-6943 . If it were only 6800 stalling, I'd assume I just messed up with lock grabbing order, but with both 2.1 branches failing, I think you are correct.
          Hide
          benedict Benedict added a comment -

          Definitely agree for the counter-cache-enabled runs, but the non-counter-cache runs have a very smooth degradation, which doesn't really fit. Still plausible, so let's re-run once the fix has been applied.

          Show
          benedict Benedict added a comment - Definitely agree for the counter-cache-enabled runs, but the non-counter-cache runs have a very smooth degradation, which doesn't really fit. Still plausible, so let's re-run once the fix has been applied.
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          Could be that 6880 in its previous iteration wasn't name-spacing cells by partition key when calling bulkGet(), and since the cell name is the same in these runs.. yeah, most likely that. Fixed in the latest CASSANDRA-6880 iteration.

          Show
          iamaleksey Aleksey Yeschenko added a comment - Could be that 6880 in its previous iteration wasn't name-spacing cells by partition key when calling bulkGet(), and since the cell name is the same in these runs.. yeah, most likely that. Fixed in the latest CASSANDRA-6880 iteration.
          Show
          iamaleksey Aleksey Yeschenko added a comment - Russ Hatch can you please rerun with the following three branches? 1. https://github.com/iamaleksey/cassandra/tree/cassandra-2.0 2. https://github.com/iamaleksey/cassandra/tree/cassandra-2.1 3. https://github.com/iamaleksey/cassandra/tree/6553 Thanks!
          Show
          rhatch Russ Hatch added a comment - Here's the data comparing the branches above with the counter cache disabled: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_one_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_one_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.uber_contention_no_cache_CL_one_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_no_cache_CL_quorum_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_no_cache_CL_quorum_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_no_cache_CL_quorum_4_3.json I've got the tests with counter cache enabled running now and will post them tonight or early tomorrow.
          Show
          rhatch Russ Hatch added a comment - Counter cache enabled tests: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_CL_one_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_CL_one_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.uber_contention_CL_one_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_CL_quorum_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_CL_quorum_4_3.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_CL_quorum_4_3.json
          Hide
          iamaleksey Aleksey Yeschenko added a comment -

          Russ Hatch One last time this week, could you run QUORUM only, counter cache ON only https://github.com/iamaleksey/cassandra/tree/cassandra-2.0 vs. https://github.com/iamaleksey/cassandra/tree/6553 ? (for screenshots)

          Thanks.

          Show
          iamaleksey Aleksey Yeschenko added a comment - Russ Hatch One last time this week, could you run QUORUM only, counter cache ON only https://github.com/iamaleksey/cassandra/tree/cassandra-2.0 vs. https://github.com/iamaleksey/cassandra/tree/6553 ? (for screenshots) Thanks.
          Hide
          rhatch Russ Hatch added a comment -

          adding logs in logs.tar.gz

          These should be ordered same as the graph links above (starting with the no counter cache tests).

          Like so:
          no counter cache/low contention/2.0/write/cl.one
          no counter cache/low contention/2.0/read/cl.one
          no counter cache/low contention/2.1/write/cl.one
          no counter cache/low contention/2.1/read/cl.one
          ............
          counter cache enabled/uber contention/aleksey's patched 2.1/read/cl.quorum

          Show
          rhatch Russ Hatch added a comment - adding logs in logs.tar.gz These should be ordered same as the graph links above (starting with the no counter cache tests). Like so: no counter cache/low contention/2.0/write/cl.one no counter cache/low contention/2.0/read/cl.one no counter cache/low contention/2.1/write/cl.one no counter cache/low contention/2.1/read/cl.one ............ counter cache enabled/uber contention/aleksey's patched 2.1/read/cl.quorum
          Hide
          rhatch Russ Hatch added a comment - - edited

          Aleksey Yeschenko – here you go:
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_CL_quorum_4_4.json
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_CL_quorum_4_4.json
          http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_CL_quorum_4_4.json

          btw, if you want to change the top title or legend information for better presentation, you can update the json. Clicking a color in the legend will disable/enable that dataset.
          If you want to make changes but don't want to push them to github, you can clone the cassandra_performance repo and run it locally from the 'graph' directory with python -m SimpleHTTPServer

          Show
          rhatch Russ Hatch added a comment - - edited Aleksey Yeschenko – here you go: http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.low_contention_CL_quorum_4_4.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.high_contention_CL_quorum_4_4.json http://riptano.github.io/cassandra_performance/graph/graph.html?stats=6553.user_contention_CL_quorum_4_4.json btw, if you want to change the top title or legend information for better presentation, you can update the json. Clicking a color in the legend will disable/enable that dataset. If you want to make changes but don't want to push them to github, you can clone the cassandra_performance repo and run it locally from the 'graph' directory with python -m SimpleHTTPServer

            People

            • Assignee:
              rhatch Russ Hatch
              Reporter:
              enigmacurry Ryan McGuire
            • Votes:
              0 Vote for this issue
              Watchers:
              8 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development