[FLINK-29956] Kafka-related test infrastructure code is scattered over multiple classes/environments

• Type: Technical Debt
• Status: Open
• Priority: Minor
• Resolution: Unresolved
• Affects Version/s: 1.17.0
• Component/s: Connectors / Kafka
• Labels: auto-deprioritized-major starter
• Assignee: Unassigned
• Reporter: Matthias Pohl
• Created: 09/Nov/22 10:58
• Updated: 20/Aug/23 22:35

Description

We had a few issues with the test stability of Kafka-related tests (FLINK-24119, FLINK-29914). A workaround was to add randomness to topic identifiers. This change required touching multiple code locations that made it obvious that there is some room for improvement (see Gabor's comment comment in the related PR). We could put some effort into unifying this by providing a common test environment that provides utility methods he mentioned in his comment:

Topic creation
Topic deletion
Random name generation
AdminClient creation
Consumer creation
Producer creation

Currently, we have Kafka instances created in different locations in our test suite (e.g. KafkaSourceTestEnv, KafkaTestEnvironmentImpl, KafkaTableTestBase). One idea is to provide a JUnit5 extension for this similarly to what we do with the MiniClusterExtension for Flink. Maybe, there's already something out there that can be used/extended.

Issue Links

• Discovered while testing: FLINK-24119 KafkaITCase.testTimestamps fails due to "Topic xxx already exist"
• Discovered while testing: FLINK-29914 KafkaTableITCase.testKafkaSourceSink fails