Uploaded image for project: 'Cassandra'
  1. Cassandra
  2. CASSANDRA-9504

Odd performance numbers comparing 2.0.15 and 2.1+

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Resolved
    • Priority: Normal
    • Resolution: Fixed
    • Fix Version/s: 2.1.6, 2.2.0 rc1
    • Component/s: None
    • Labels:
      None

      Description

      I've been doing some basic performance testing of batch commitlog on 2.0.15, 2.1-head, and 2.2.-head. I have been seeing very odd results where 2.0.15 performance is incredibly high, suspiciously so, when compared to 2.1 or 2.2.
      Example graph:
      http://cstar.datastax.com/graph?stats=2ea11748-0562-11e5-9045-42010af0688f&metric=op_rate&operation=1_write&smoothing=1&show_aggregates=true&xmin=0&xmax=1570.36&ymin=0&ymax=127509.8

      I'm using the following yaml options:

      commitlog_sync: batch
      commitlog_sync_batch_window_in_ms: 50
      commitlog_sync_period_in_ms: null
      concurrent_writes: 64
      

      I am able to reproduce this on two separate clusters, physical hardware, with SSDs. I have not been able to reproduce this on gce, or a physical cluster I have access to with HDD. I can reproduce this consistently, with different stress options, and with higher data load on the nodes. I can also reproduce this myself on these machines, without using the C* perf tool to set up the cluster and perform the operations.

      When running 2.0.15 batch commitlog vs periodic, I do see a difference as expected:
      http://cstar.datastax.com/graph?stats=4ae77aac-056f-11e5-8344-42010af0688f&metric=op_rate&operation=1_write&smoothing=1&show_aggregates=true&xmin=0&xmax=68.09&ymin=0&ymax=166966.8

      I am using the newest stress code on trunk. What should I be looking for to explain the anomalous performance here?

        Attachments

        1. jstack.tar.gz
          186 kB
          Philip Thompson
        2. 9504.txt
          3 kB
          Jonathan Ellis

          Activity

            People

            • Assignee:
              jbellis Jonathan Ellis
              Reporter:
              philipthompson Philip Thompson
              Authors:
              Jonathan Ellis
              Reviewers:
              Benedict Elliott Smith
            • Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: