[FLINK-22136] Device application for unaligned checkpoint test on cluster

• Type: Sub-task
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 1.14.0
• Labels: pull-request-available
• Assignee: Anton Kalashnikov
• Reporter: Arvid Heise
• Created: 07/Apr/21 09:25
• Updated: 22/Jun/21 14:06
• Resolved: 27/Apr/21 08:08

Description

To test unaligned checkpoints, we should use a few different applications that use different features:

• Mixing forward/rescale channels with keyby or other shuffle operations
• Unions
• 2 or n-ary operators
• Associated state ((keyed) process function)
• Correctness verifications

The sinks should not be mocked but rather should be able to induce a fair amount of backpressure into the system. Quite possibly, it would be a good idea to have a way to add more backpressure to the sink by running the respective system on the cluster and be able to add/remove parallel instances.

Things to check in the application

• Inflight data is restored to the correct keygroups -> can be checked with keyed state in a process function
• Correctness: Completeness (no lost records) + no duplicates
• Orderness of data for keyed exchanges (we guarantee that records with the same key retain orderness across keyed operators)
• (To detect errors early, we can also use magic headers)

Issue Links

• links to: GitHub Pull Request #15598