[FLINK-25249] Reimplement KafkaSourceTestEnv with KafkaContainer

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.14.0, 1.13.3
• Fix Version/s: 1.15.0
• Component/s: Connectors / Kafka
• Labels: pull-request-available
• Assignee: Qingsheng Ren
• Reporter: Qingsheng Ren
• Created: 10/Dec/21 09:07
• Updated: 25/Feb/22 15:46
• Resolved: 25/Feb/22 15:46

Description

Currently KafkaSourceTestEnv is based on KafkaTestBase, which is quite flaky (probably because of the NetworkFailureProxy) and complex. Using KafkaContainer instead could simplify and stabilize test cases of KafkaSource

Issue Links

• fixes: FLINK-23391 KafkaSourceReaderTest.testKafkaSourceMetrics fails on azure
• fixes: FLINK-20928 KafkaSourceReaderTest.testOffsetCommitOnCheckpointComplete:189->pollUntil:270 » Timeout
• is depended upon by: FLINK-23956 KafkaITCase.testOneSourceMultiplePartitions fails due to "The topic metadata failed to propagate to Kafka broker"
• is depended upon by: FLINK-16908 FlinkKafkaProducerITCase testScaleUpAfterScalingDown Timeout expired while initializing transactional state in 60000ms.
• is depended upon by: FLINK-24119 KafkaITCase.testTimestamps fails due to "Topic xxx already exist"
• links to: GitHub Pull Request #18246
• links to: GitHub Pull Request #18656