We have a system test suite (see http://wiki.apache.org/cassandra/HowToContribute ) but it only runs against the local machine, which is suboptimal even if we didn't have a bunch of optimizations to make writes to a local node skip most of the messaging layer.
We'd like to automate testing against a whole cluster. Paul has some code using libcloud to do stress tests against an ephemeral rackspace cloud servers cluster at http://github.com/pquerna/ccb, which is probably useful as a starting point, but rather than running stress.py you'd want to put together a more comprehensive suite, based on the existing system tests but also explicitly covering things like range queries that span multiple nodes and failure conditions like Hinted Handoff (see http://wiki.apache.org/cassandra/HintedHandoff ) and repair (http://wiki.apache.org/cassandra/ArchitectureAntiEntropy ) resulting from deliberately disabling a node temporarily.
Adding tests for ops tasks like http://wiki.apache.org/cassandra/Operations would be awesome too.
CASSANDRA-561 might make something like this possible using a simulator instead of hardware-level virtualization, which would be both more convenient and less expensive. But someone just dumped a huge zip in our laps w/o really trying to collaborate with us there, so if you want to try to resurrect that code or otherwise go that route, be more open about what you're doing.