[FLINK-11038] Rewrite Kafka at-least-once it cases

• Type: Bug
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: 1.7.0
• Component/s: Connectors / Kafka
• Labels: auto-deprioritized-major stale-minor
• Assignee: Unassigned
• Reporter: Piotr Nowojski
• Created: 30/Nov/18 08:55
• Updated: 22/Nov/21 17:01

Description

Currently they are using NetworkFailuresProxy which is unstable both for Kafka 0.11 in exactly once mode (in 50% tests are live locking) and for Kafka 2.0 (and because of that currently testOneToOneAtLeastOnceRegularSink and testOneToOneAtLeastOnceCustomOperator tests are disabled).

Those tests should either be rewritten to SIGKILL Flink's process doing the writing. Either as an ITCase SIGKILL-ing task manager or test harness SIGKILL-ing/exiting test harness process.

We can not simply use test harness and do not close it to simulate failure, because we want to make sure that we have flushed the records during checkpoint. If we do not SIGKILL the process, the background Kafka client's threads can just send those records for us.

Issue Links

• supercedes: FLINK-10838 Rewrite Kafka tests that fail Kafka brokers