Details

    • Type: Test Test
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Fix Version/s: None
    • Component/s: Tools
    • Labels:

      Description

      We have a stress test in contrib/py_stress, and Paul has a tool using libcloud to automate running it against an ephemeral cluster of rackspace cloud servers, but to really qualify as "performance regression tests" we need to

      • test a wide variety of data types (skinny rows, wide rows, different comparator types, different value byte[] sizes, etc)
      • produce pretty graphs. seriously.
      • archive historical data somewhere for comparison (rackspace can provide a VM to host a db for this, if the ASF doesn't have something in place for this kind of thing already)

        Activity

        Hide
        Kushal Pisavadia added a comment -

        I'm pretty interested by this problem, but I see a few different directions that it could be taken in.

        Would this tool be used as part of the continuous integration process? If so, is it aimed at entire coverage or just basic regression tests to make sure new feature <x> hasn't caused too much of a problem? Would it take into account different configurations for read/write heavy nodes?

        Would there be default datasets to keep things reproducible or would you just define lengths of data types and fill on the fly?

        A lot of the performance comparisons you see don't take into account specific features of the databases and just go by lowest common denominator. How generic should the performance test be?

        Finally, I've been trying to find research related to performance testing these new(er) database systems and have only come across the works of Brian Cooper to be useful. If you could provide any other sources that have been successful It would be much appreciated.

        Show
        Kushal Pisavadia added a comment - I'm pretty interested by this problem, but I see a few different directions that it could be taken in. Would this tool be used as part of the continuous integration process? If so, is it aimed at entire coverage or just basic regression tests to make sure new feature <x> hasn't caused too much of a problem? Would it take into account different configurations for read/write heavy nodes? Would there be default datasets to keep things reproducible or would you just define lengths of data types and fill on the fly? A lot of the performance comparisons you see don't take into account specific features of the databases and just go by lowest common denominator. How generic should the performance test be? Finally, I've been trying to find research related to performance testing these new(er) database systems and have only come across the works of Brian Cooper to be useful. If you could provide any other sources that have been successful It would be much appreciated.
        Hide
        Jonathan Ellis added a comment -

        > Would this tool be used as part of the continuous integration process?

        The easier it can be integrated w/ Hudson, the more likely this is to happen. But I would rather spend more time on making a useful tool at first, than making it Hudson-able, if there is tension between those goals.

        > If so, is it aimed at entire coverage or just basic regression tests to make sure new feature <x> hasn't caused too much of a problem? Would it take into account different configurations for read/write heavy nodes?

        "Yes."

        The right approach is to get something simple working and add features as time permits.

        > How generic should the performance test be?

        It should be Cassandra-specific, not generic to other databases.

        > If you could provide any other sources that have been successful It would be much appreciated.

        There's vpork, a voldemort benchmark, but my impression is it's rather less sophisticated than our own stress.py.

        Brian has indicated that he wants to OSS his stuff, but lawyers are getting in the way; he's probably willing to offer tips if you contact him.

        Show
        Jonathan Ellis added a comment - > Would this tool be used as part of the continuous integration process? The easier it can be integrated w/ Hudson, the more likely this is to happen. But I would rather spend more time on making a useful tool at first, than making it Hudson-able, if there is tension between those goals. > If so, is it aimed at entire coverage or just basic regression tests to make sure new feature <x> hasn't caused too much of a problem? Would it take into account different configurations for read/write heavy nodes? "Yes." The right approach is to get something simple working and add features as time permits. > How generic should the performance test be? It should be Cassandra-specific, not generic to other databases. > If you could provide any other sources that have been successful It would be much appreciated. There's vpork, a voldemort benchmark, but my impression is it's rather less sophisticated than our own stress.py. Brian has indicated that he wants to OSS his stuff, but lawyers are getting in the way; he's probably willing to offer tips if you contact him.
        Hide
        Jeremy Hanna added a comment - - edited

        As previously referenced, the YCSB (Brian Cooper's) benchmark is generic but is now open source so it could be used for trending and/or comparison against other similar data stores.

        http://wiki.github.com/brianfrankcooper/YCSB/

        Show
        Jeremy Hanna added a comment - - edited As previously referenced, the YCSB (Brian Cooper's) benchmark is generic but is now open source so it could be used for trending and/or comparison against other similar data stores. http://wiki.github.com/brianfrankcooper/YCSB/
        Hide
        Johan Oskarsson added a comment -

        I should have replied here earlier, sent my thoughts as an email to the mailing list instead a while back.

        I am working on parts of a Cassandra performance testing solution when I have time. This is my current thinking:

        • The benchmarking would be done by YCSB as suggested. It is known to work and Brian has been very helpful in accepting changes. I have added support for exporting the results into a JSON file so that it can be read programmatically.
        • Since getting a hold of servers to run this on I suggest starting out by using one of the cloud services, hopefully Rackspace. While there are a few scripts bundles out there to deploy Cassandra clusters on these I have started adding support for Cassandra to Whirr. That project intends to add support for many of the popular Apache projects (Hadoop, HBase, Zookeeper, Cassandra etc). As an Apache project with developers from Cloudera, Yahoo!, HP etc Whirr will hopefully get continued support. Especially the inclusion of HBase would make it an interesting choice. Sharing a similar test setup to them would be beneficial to both projects. http://incubator.apache.org/whirr/
        • In order to make sense of the YCSB output I have forked a Hudson plugin and ported it to read the YCSB output files. It is still very much a rough first version, but would eventually provide reports, graphs and alerts if the performance drops unexepectedly. http://github.com/johanoskarsson/hudson-ycsb

        Thoughts on this approach?

        Show
        Johan Oskarsson added a comment - I should have replied here earlier, sent my thoughts as an email to the mailing list instead a while back. I am working on parts of a Cassandra performance testing solution when I have time. This is my current thinking: The benchmarking would be done by YCSB as suggested. It is known to work and Brian has been very helpful in accepting changes. I have added support for exporting the results into a JSON file so that it can be read programmatically. Since getting a hold of servers to run this on I suggest starting out by using one of the cloud services, hopefully Rackspace. While there are a few scripts bundles out there to deploy Cassandra clusters on these I have started adding support for Cassandra to Whirr. That project intends to add support for many of the popular Apache projects (Hadoop, HBase, Zookeeper, Cassandra etc). As an Apache project with developers from Cloudera, Yahoo!, HP etc Whirr will hopefully get continued support. Especially the inclusion of HBase would make it an interesting choice. Sharing a similar test setup to them would be beneficial to both projects. http://incubator.apache.org/whirr/ In order to make sense of the YCSB output I have forked a Hudson plugin and ported it to read the YCSB output files. It is still very much a rough first version, but would eventually provide reports, graphs and alerts if the performance drops unexepectedly. http://github.com/johanoskarsson/hudson-ycsb Thoughts on this approach?
        Hide
        Jonathan Ellis added a comment -

        working on this out of tree, similar to dtests

        Show
        Jonathan Ellis added a comment - working on this out of tree, similar to dtests

          People

          • Assignee:
            Tyler Patterson
            Reporter:
            Jonathan Ellis
          • Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development