[STORM-2008] kafka jars must be part of the storm-kafka topology

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Invalid
• Component/s: storm-kafka
• Assignee: Unassigned
• Reporter: Raghav Kumar Gautam
• Created: 26/Jul/16 22:53
• Updated: 26/Jul/16 23:12
• Resolved: 26/Jul/16 23:12

Description

As of now the dependency is present but it's scope is provided, this is incorrect.
https://github.com/apache/storm/blob/master/external/storm-kafka/pom.xml#L121

        <dependency>
            <groupId>org.apache.kafka</groupId>
            <artifactId>${storm.kafka.artifact.id}</artifactId>
            <version>${storm.kafka.version}</version>
            <scope>provided</scope>
        </dependency>

This leads to following stacktrace in worker, when we try to run TridentKafkaWordCount topology.

    This is for solving following exception:
    Exception in thread "main" java.lang.NoClassDefFoundError: kafka/api/OffsetRequest
        at org.apache.storm.kafka.KafkaConfig.<init>(KafkaConfig.java:48)
        at org.apache.storm.kafka.trident.TridentKafkaConfig.<init>(TridentKafkaConfig.java:30)
        at org.apache.storm.starter.trident.TridentKafkaWordCount.createTransactionalKafkaSpout(TridentKafkaWordCount.java:102)
        at org.apache.storm.starter.trident.TridentKafkaWordCount.addTridentState(TridentKafkaWordCount.java:131)
        at org.apache.storm.starter.trident.TridentKafkaWordCount.buildConsumerTopology(TridentKafkaWordCount.java:149)
        at org.apache.storm.starter.trident.TridentKafkaWordCount.runMain(TridentKafkaWordCount.java:249)
        at org.apache.storm.starter.trident.TridentKafkaWordCount.main(TridentKafkaWordCount.java:240)
    Caused by: java.lang.ClassNotFoundException: kafka.api.OffsetRequest
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        ... 7 more

Issue Links

• is related to: STORM-1876 Provide option to build storm-kafka against different kafka clients
• links to: GitHub Pull Request #1596