[CASSANDRA-9007] Run stress nightly against trunk in a way that validates

• Type: Task
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Component/s: Feature/Lightweight Transactions
• Labels: LWT monthly-release
• Assignee: Philip Thompson
• Reporter: Ariel Weisberg

Authors

Philip Thompson

• Created: 19/Mar/15 21:50
• Updated: 25/Oct/19 13:11
• Resolved: 27/Apr/16 18:43

Description

Stress has some very basic validation functionality when used without workload profiles. It found a bug on trunk when I first ran it so it has value even though the validation is basic.

As a beachhead for the kind of blackbox validation that we are missing we can start by running stress nightly or 24/7 in some rotation.

There should be two jobs. One job has inverted success criteria (C* should lose some data) and the job should only "pass" if the failure is detected. This is just to prove that the harness reports failure if failure occurs.

Another would be the real job that runs stress, parses and parses the output for reports of missing data.

This job is the first pass and basis of what we can point to when a developer makes a change, implements a feature, or fixes a bug, and say "go add validation to this job."

Follow on tickets to link to this

• Test multiple configurations
• Get stress to validate more query functionality and APIs (counters, LWT, batches)
• Parse logs and fail tests on error level logs (great way to improve log messages over time)
• ?

I am going to hold off on creating a ton of issues until we have a basic version of the job running.

Issue Links

• blocks: CASSANDRA-9042 Make trunk always releasable

Sub-Tasks

• Build prototype for validation testing harness: Resolved
• Run validation harness against a real cluster: Resolved
• Expand harness to test against all compaction strategies: Resolved
• Validate that the Java driver and server don't generate timeouts in spuriously: Open