Details

    • Type: Test Test
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Duplicate
    • Fix Version/s: None
    • Component/s: Tools
    • Labels:

      Description

      We have a system test suite (see http://wiki.apache.org/cassandra/HowToContribute ) but it only runs against the local machine, which is suboptimal even if we didn't have a bunch of optimizations to make writes to a local node skip most of the messaging layer.

      We'd like to automate testing against a whole cluster. Paul has some code using libcloud to do stress tests against an ephemeral rackspace cloud servers cluster at http://github.com/pquerna/ccb, which is probably useful as a starting point, but rather than running stress.py you'd want to put together a more comprehensive suite, based on the existing system tests but also explicitly covering things like range queries that span multiple nodes and failure conditions like Hinted Handoff (see http://wiki.apache.org/cassandra/HintedHandoff ) and repair (http://wiki.apache.org/cassandra/ArchitectureAntiEntropy ) resulting from deliberately disabling a node temporarily.

      Adding tests for ops tasks like http://wiki.apache.org/cassandra/Operations would be awesome too.

      Finally, CASSANDRA-561 might make something like this possible using a simulator instead of hardware-level virtualization, which would be both more convenient and less expensive. But someone just dumped a huge zip in our laps w/o really trying to collaborate with us there, so if you want to try to resurrect that code or otherwise go that route, be more open about what you're doing.

        Issue Links

          Activity

          Hide
          Jeremy Hanna added a comment - - edited

          As another data point, Voldemort uses EC2 to perform multi-node system tests. The code they use along with documentation is found here http://wiki.github.com/voldemort/voldemort/ec2-testing-infrastructure Perhaps there are some general ideas there for what and how to test at that level. See also http://wiki.github.com/voldemort/voldemort/multi-datacenter-results

          Show
          Jeremy Hanna added a comment - - edited As another data point, Voldemort uses EC2 to perform multi-node system tests. The code they use along with documentation is found here http://wiki.github.com/voldemort/voldemort/ec2-testing-infrastructure Perhaps there are some general ideas there for what and how to test at that level. See also http://wiki.github.com/voldemort/voldemort/multi-datacenter-results
          Hide
          Peter Tillotson added a comment -

          I think that this may already be largely possible via a virtual network, though it presents an ideal case. Libvirt on Linux allows you to set up virtual network interfaces on a given host so you could get multiple instances of Cassandra running on the same local hardware, one could probably simulate failing nodes.
          see: virsh --net-create for examples

          Show
          Peter Tillotson added a comment - I think that this may already be largely possible via a virtual network, though it presents an ideal case. Libvirt on Linux allows you to set up virtual network interfaces on a given host so you could get multiple instances of Cassandra running on the same local hardware, one could probably simulate failing nodes. see: virsh --net-create for examples
          Hide
          Jonathan Ellis added a comment -
          Show
          Jonathan Ellis added a comment - see CASSANDRA-1859

            People

            • Assignee:
              Unassigned
              Reporter:
              Jonathan Ellis
            • Votes:
              1 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development