Details

    • Type: Test
    • Status: Resolved
    • Priority: Major
    • Resolution: Fixed
    • Fix Version/s: 0.7.1
    • Component/s: Tools
    • Labels:
      None

      Description

      Distributed Test Harness

      • deploys a cluster on a cloud provider
      • runs tests targeted at the cluster
      • tears down the cluster

        Issue Links

          Activity

          Hide
          hudson Hudson added a comment -

          Integrated in Cassandra-0.7 #151 (See https://hudson.apache.org/hudson/job/Cassandra-0.7/151/)

          Show
          hudson Hudson added a comment - Integrated in Cassandra-0.7 #151 (See https://hudson.apache.org/hudson/job/Cassandra-0.7/151/ )
          Hide
          brandon.williams Brandon Williams added a comment -

          Committed.

          Show
          brandon.williams Brandon Williams added a comment - Committed.
          Hide
          stuhood Stu Hood added a comment -

          Attaching a rebase for the 0.7 branch.

          Show
          stuhood Stu Hood added a comment - Attaching a rebase for the 0.7 branch.
          Hide
          brandon.williams Brandon Williams added a comment - - edited

          Looks good to me, as a beginning. I'd like to see the existing tests improved (there should at least be constant reads during bootstrap/move, since we've had bugs there before) but this is good enough to move forward with. I'm prepared to commit after WHIRR-155 is done.

          Show
          brandon.williams Brandon Williams added a comment - - edited Looks good to me, as a beginning. I'd like to see the existing tests improved (there should at least be constant reads during bootstrap/move, since we've had bugs there before) but this is good enough to move forward with. I'm prepared to commit after WHIRR-155 is done.
          Hide
          kingryan Ryan King added a comment -

          Add another test for writing with one and reading with all (the last of the strong consistency scenarios).

          Show
          kingryan Ryan King added a comment - Add another test for writing with one and reading with all (the last of the strong consistency scenarios).
          Hide
          stuhood Stu Hood added a comment - - edited

          This should be ready for review. It uses Whirr to deploy a 4 node cluster in a compute provider, pushes artifacts from the working copy to a blobstore, and then executes JUnit tests. See test/distributed/README.txt for more details.

          There are a few obvious improvements that would be nice to have, including:

          • Collecting logs from the test nodes
          • Reusing an existing cluster
          • Launching client nodes (for performance, rather than correctness)

          In order to specify the tarball URL to install Cassandra from, we needed to make a modification to Whirr, which should be in Whirr 0.3.0: see WHIRR-155. Until 0.3.0 is released, patch 0002 adds maven.twttr.com to our Ivy config, to pull in a version of Whirr that includes 155. Also, when 0.3.0 is released, we'll be able to switch from hoodidge.net to Whirr's sanctioned runurl base.

          Show
          stuhood Stu Hood added a comment - - edited This should be ready for review. It uses Whirr to deploy a 4 node cluster in a compute provider, pushes artifacts from the working copy to a blobstore, and then executes JUnit tests. See test/distributed/README.txt for more details. There are a few obvious improvements that would be nice to have, including: Collecting logs from the test nodes Reusing an existing cluster Launching client nodes (for performance, rather than correctness) In order to specify the tarball URL to install Cassandra from, we needed to make a modification to Whirr, which should be in Whirr 0.3.0: see WHIRR-155 . Until 0.3.0 is released, patch 0002 adds maven.twttr.com to our Ivy config, to pull in a version of Whirr that includes 155. Also, when 0.3.0 is released, we'll be able to switch from hoodidge.net to Whirr's sanctioned runurl base.
          Hide
          stuhood Stu Hood added a comment -

          > It would be nice to not require ec2/rax to run the test suite.
          Theoretically, any cloud supported by jclouds will work to run these tests, libvirt included. See http://code.google.com/p/jclouds/#Supported_Services

          Show
          stuhood Stu Hood added a comment - > It would be nice to not require ec2/rax to run the test suite. Theoretically, any cloud supported by jclouds will work to run these tests, libvirt included. See http://code.google.com/p/jclouds/#Supported_Services
          Hide
          jbellis Jonathan Ellis added a comment -

          On the old CASSANDRA-874, Peter pointed out that "Libvirt on Linux allows you to set up virtual network interfaces on a given host so you could get multiple instances of Cassandra running on the same local hardware, one could probably simulate failing nodes. see: virsh --net-create for examples"

          It would be nice to not require ec2/rax to run the test suite.

          Show
          jbellis Jonathan Ellis added a comment - On the old CASSANDRA-874 , Peter pointed out that "Libvirt on Linux allows you to set up virtual network interfaces on a given host so you could get multiple instances of Cassandra running on the same local hardware, one could probably simulate failing nodes. see: virsh --net-create for examples" It would be nice to not require ec2/rax to run the test suite.
          Hide
          stuhood Stu Hood added a comment -

          Spent the day writing a test against 0.7.0-rc2, and accidentally reproduced CASSANDRA-1829. Good start!

          Still planning to post a patch this week: a few subtasks to wrap up first.

          Show
          stuhood Stu Hood added a comment - Spent the day writing a test against 0.7.0-rc2, and accidentally reproduced CASSANDRA-1829 . Good start! Still planning to post a patch this week: a few subtasks to wrap up first.
          Hide
          stuhood Stu Hood added a comment -

          Note that the branch linked above relies on non-standard Whirr scripts to deploy Cassandra (hence the requirement to use hoodidge.net/scripts as the runurl base). We're going to be submitting these modified scripts to Whirr this week.

          Show
          stuhood Stu Hood added a comment - Note that the branch linked above relies on non-standard Whirr scripts to deploy Cassandra (hence the requirement to use hoodidge.net/scripts as the runurl base). We're going to be submitting these modified scripts to Whirr this week.
          Hide
          kelvin Kelvin Kakugawa added a comment -

          The above branch targets 0.7, however, it can be backported to 0.6.

          If anyone is interested in testing 0.6, I would refer you to Stu Hood for advice on setting up the node setup scripts.

          Show
          kelvin Kelvin Kakugawa added a comment - The above branch targets 0.7, however, it can be backported to 0.6. If anyone is interested in testing 0.6, I would refer you to Stu Hood for advice on setting up the node setup scripts.
          Hide
          jeromatron Jeremy Hanna added a comment - - edited

          As we were talking 0.7 testing, Jonathan mentioned that moving nodes would be a good candidate for testing.

          Show
          jeromatron Jeremy Hanna added a comment - - edited As we were talking 0.7 testing, Jonathan mentioned that moving nodes would be a good candidate for testing.
          Hide
          kelvin Kelvin Kakugawa added a comment -

          The above list is not comprehensive.

          Show
          kelvin Kelvin Kakugawa added a comment - The above list is not comprehensive.
          Hide
          kelvin Kelvin Kakugawa added a comment -

          Scenarios that should be tested:

          • get range slices - test boundaries, walking
          • adding a node
          • bootstrap / adding multiple nodes simultaneously
          • redistributing keys
          • compaction/cleanup/repair + integrity check
          • hinted handoff
          • read repair
          • net splits / failures
          • drive failures (wipe/kill one dir)

          and, permutations of the above--simulatenous and sequential.

          Show
          kelvin Kelvin Kakugawa added a comment - Scenarios that should be tested: get range slices - test boundaries, walking adding a node bootstrap / adding multiple nodes simultaneously redistributing keys compaction/cleanup/repair + integrity check hinted handoff read repair net splits / failures drive failures (wipe/kill one dir) and, permutations of the above--simulatenous and sequential.
          Hide
          kelvin Kelvin Kakugawa added a comment -

          latest code is here:
          https://github.com/kakugawa/cassandra/tree/distributed-test-harness

          A patch will be cut, later.

          Show
          kelvin Kelvin Kakugawa added a comment - latest code is here: https://github.com/kakugawa/cassandra/tree/distributed-test-harness A patch will be cut, later.

            People

            • Assignee:
              kelvin Kelvin Kakugawa
              Reporter:
              kelvin Kelvin Kakugawa
              Reviewer:
              Brandon Williams
            • Votes:
              2 Vote for this issue
              Watchers:
              9 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development