[KAFKA-10202] SmokeTest should provide a machanism to generate continuously AND verify the result

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Component/s: streams, system tests
• Assignee: Unassigned
• Reporter: John Roesler
• Created: 26/Jun/20 02:34
• Updated: 26/Jun/20 02:34

Description

Several system tests use the SmokeTestDriver, but they have to choose between generating for a fixed period of time (say, two minutes) before verifying results OR generating until cancelled with no verification.

It's not impossible to implement "generate until cancelled and then verify", and doing so would both speed up tests that are just blocked by the two-minute generation period AND decrease flakiness in other tests, when they can't complete all their operations within the deadline before verification begins.

One ides is to revamp the smoke test driver's verification logic to make it consume the input and determine the expected results, so that the verifier can be run as a separate process instead of passing the expected results straight from the generator to the verifier. This is how the RelationalSmokeTest works.

An alternative is to register a signal handler so that we can signal the driver to stop generating and start verifying.