[CASSANDRA-18451] CEP-15: (C*) Improve the chaos generation for Burn Tests: slow/flakey connections and dropped messages

• Type: Improvement
• Status: Resolved
• Priority: Normal
• Resolution: Fixed
• Fix Version/s: NA
• Component/s: Accord
• Labels: pull-request-available
• Change Category: Quality Assurance
• Complexity: Normal
• Platform: All

Source Control Link

https://github.com/apache/cassandra-accord/commit/fc3e1af12554e3befe6e44f4664278d91b4c0415

Test and Documentation Plan

new tests

• Assignee: David Capwell
• Reporter: David Capwell

Authors

David Capwell

Reviewers

Benedict Elliott Smith

• Created: 14/Apr/23 19:51
• Updated: 27/Oct/23 15:41
• Resolved: 13/Sep/23 22:53

Description

Burn test should be enhanced to add the following:

1) message dropping from one node to another (this is different than the current partition set logic in accord.impl.basic.Cluster#partitionSet)
2) for messages with callbacks, trigger failure case
3) redundant message delivery

Related work:

• Simulator’s org.apache.cassandra.simulator.systems.SimulatedAction#applyToMessage

• Figures out what delivery action to perform via org.apache.cassandra.simulator.FutureActionScheduler#shouldDeliver
  • timeout if dropPartition[from] != dropPartition[to] // either to/from is in drop partition, but not both
  • config asked to override and deliver
  • 50/50 chance to deliver, after that 50/50 to deliver w/ timeout, after that cause a failure
• in C* failure is an enum with Timeout and Unknown
• knows the schedule time and the message expire time, and can promote a DELIVER event to DELIVER_AND_TIMEOUT
• triggers the timeout

Issue Links

• links to: CI: trunk
• links to: GH: trunk
• links to: GitHub Pull Request #43