Uploaded image for project: 'Apache Storm'
  1. Apache Storm
  2. STORM-967

Unable to package storm-kafka & storm-hdfs connectors in a topology

    XMLWordPrintableJSON

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Critical
    • Resolution: Fixed
    • Affects Version/s: None
    • Fix Version/s: 0.10.0, 1.0.0
    • Component/s: storm-hdfs, storm-kafka
    • Labels:
      None

      Description

      Based on my testing storm-kafka & storm-hdfs connector along with it dependencies are unable to be packaged together .

      If we tried to package it using maven assembly plugin storm-hdfs errors out saying
      "java.lang.RuntimeException: Error preparing HdfsBolt: No FileSystem for scheme: hdfs"

      To avoid we need to use maven shade plugin as documented in the README here
      https://github.com/apache/storm/tree/master/external/storm-hdfs

      Now when we use maven shade plugin storm-kafka throws an initial error about

         java.lang.NoClassDefFoundError: Could not initialize class com.yammer.metrics.Metrics
      at kafka.metrics.KafkaMetricsGroup$class.newTimer(KafkaMetricsGroup.scala:52) at kafka.consumer.FetchRequestAndResponseMetrics.newTimer(FetchRequestAndResponseStats.scala:25)
      at kafka.consumer.FetchRequestAndResponseMetrics.<init>(FetchRequestAndResponseStats.scala:26)
      at kafka.consumer.FetchRequestAndResponseStats.<init>(FetchRequestAndResponseStats.scala:37)
      

      After we explicitly added metrics dependency to the pom file
      KafkaSpout now errors out with
      IllegalStateException KafkaMetricsGroup.newTimer

      KafkaSpout works fine if we use assembly plugin. Due to the behavior of these two connectors one works assembly and another works with shade it looks like we won't be able to use them together.

        Attachments

          Activity

            People

            • Assignee:
              revans2 Robert Joseph Evans
              Reporter:
              sriharsha Harsha
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: