[KAFKA-9992] EmbeddedKafkaCluster.deleteTopicAndWait not working with kafka_2.13

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.4.1
• Fix Version/s: 2.6.0, 2.4.2, 2.5.1
• Component/s: packaging, streams
• Assignee: Andras Katona
• Reporter: Andras Katona
• Created: 14/May/20 13:07
• Updated: 19/May/20 19:09
• Resolved: 19/May/20 18:55

Description

Kafka Streams artifact is depending on kafka_2.12 as of now, it is in the kafka-streams-2.4.1.pom:

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.12</artifactId>
  <version>2.4.1</version>
  <scope>test</scope>
</dependency>  

But it is not hardcoded, whatever scala version was used to compile this component before uploading, that will be present in the pom.

When I'm using these deps:

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka-streams</artifactId>
  <version>2.4.1</version>
  <classifier>test</classifier>
  <scope>test</scope>
</dependency>

<dependency>
  <groupId>org.apache.kafka</groupId>
  <artifactId>kafka_2.13</artifactId>
  <version>2.4.1</version>
  <classifier>test</classifier>
  <scope>test</scope>
</dependency>

My test fails with the following exception (deleteTopicAndWait is called in my @After method):

java.lang.NoSuchMethodError: scala.collection.JavaConverters.setAsJavaSetConverter(Lscala/collection/Set;)Lscala/collection/convert/Decorators$AsJava;
        at org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster$TopicsDeletedCondition.conditionMet(EmbeddedKafkaCluster.java:316)
        at org.apache.kafka.test.TestUtils.lambda$waitForCondition$4(TestUtils.java:370)
        at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:417)
        at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:385)
        at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:368)
        at org.apache.kafka.test.TestUtils.waitForCondition(TestUtils.java:356)
        at org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.deleteTopicsAndWait(EmbeddedKafkaCluster.java:266)
        at org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster.deleteTopicAndWait(EmbeddedKafkaCluster.java:221)

I modified kafka build locally to separate artifacts based on scala version just like it is done with kafka core, and I pulled in kafka-streams_2.13 from my local mvn repo and test was working again.

I was only trying with 2.4.1, but I'm assuming other versions are also affected, please add the proper versions and proper components too (in case it's not packaging).

Issue Links

• links to: GitHub Pull Request #8673