[KAFKA-14344] Improve MM2 integration test by building EmbeddedKafkaCluster with common configs used for all clients

• Type: Improvement
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 3.4.0
• Component/s: connect, mirrormaker, unit tests
• Assignee: Omnia Ibrahim
• Reporter: Omnia Ibrahim
• Due: 31/Oct/22
• Created: 31/Oct/22 15:53
• Updated: 07/Nov/22 10:19
• Resolved: 07/Nov/22 10:19

Description

Connect and MirrorMaker's integration test use `EmbeddedKafkaCluster` and `EmbeddedConnectCluster` to setup connect cluster during testing. Both classes are easy to setup if the test needs vanilla clusters, however, it's a lot of work to make it set it up with more advanced config (for example authentication and authorization) where admin, consumer and producer clients need more configuration. 

1. I am proposing adding extra parameter `additionalClientConfigs` to `EmbeddedKafkaCluster` constructor. The new parameter will be used

• Setup Producer Client in `EmbeddedKafkaCluster.doStart` which is initializing `producer` client that is used in `EmbeddedKafkaCluster.produce`

• Setup Producer Client in `EmbeddedKafkaCluster.createProducer` used in `EmbeddedKafkaCluster.transactionalProducer`

• Setup Admin Client in `EmbeddedKafkaCluster.createAdminClient` used in `EmbeddedKafkaCluster.createTopic`, `EmbeddedKafkaCluster.consumeAll`, `EmbeddedKafkaCluster.describeTopics` and `EmbeddedKafkaCluster.deleteTopic`

• Setup Consumer Client in `EmbeddedKafkaCluster.createConsumer` used in `EmbeddedKafkaCluster.createConsumerAndSubscribeTo` and `EmbeddedKafkaCluster.consumeAll`

2. And add `EmbeddedConnectCluster.Builder.additionalKafkaClusterClientConfigs`.

 

Tests impacted by this 

• MirrorMaker integration tests
• `org.apache.kafka.connect.integration.ExactlyOnceSourceIntegrationTest`

Issue Links

• links to: GitHub Pull Request #12804