[FLINK-20103] Improve test coverage with chaos testing & side-by-side tests

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Runtime / Checkpointing, Runtime / Network, Runtime / State Backends, Tests
• Labels: auto-deprioritized-major auto-deprioritized-minor pull-request-available
• Assignee: Unassigned
• Reporter: Roman Khachatryan
• Created: 12/Nov/20 08:56
• Updated: 24/Aug/23 22:35

Description

This is a follow-up ticket after FLINK-20097.

With the current setup (UnalignedITCase):

• race conditions are not detected reliably (1 per tens of runs)
• require changing the configuration (low checkpoint timeout)
• adding a new job graph often reveals a new bug

An additional issue with the current setup is that it's difficult to git bisect (for long ranges). 

Changes that might hide the bugs:

• having Preconditions in ChannelStatePersister (slow down processing)
• some Preconditions may mask errors by causing job restart
• timings in tests (UnalignedITCase)

 Some options to consider

1. chaos monkey tests including induced latency and/or CPU bursts - on different workloads/configs
2. side-by-side tests with randomized inputs/configs

Extending Jepsen coverage further (validating output) does not seem promising in the context of Flink because it's output isn't linearisable.
 

Some tools for (1) that could be used:

1. https://github.com/chaosblade-io/chaosblade (docs need translation)
2. https://github.com/Netflix/chaosmonkey - requires spinnaker (CD)

3. jvm agent: https://github.com/mrwilson/byte-monkey
4. https://vmware.github.io/mangle/ - supports java method latency; ui oriented?; not actively maintained?

 
 

Issue Links

• is cloned by: FLINK-22232 Improve test coverage for network stack
• is related to: FLINK-25842 [v2] FLIP-158: Generalized incremental checkpoints
• relates to: FLINK-20097 Race conditions in InputChannel.ChannelStatePersister
• relates to: FLINK-20079 Modified UnalignedCheckpointITCase...MassivelyParallel fails
• relates to: FLINK-8971 Create general purpose testing job
• relates to: FLINK-19520 Add reliable test randomization for checkpointing
• links to: GitHub Pull Request #15631