Flume
  1. Flume
  2. FLUME-2392

Flume MorphlineSink can not work because of java.lang.NoClassDefFoundError: org/kitesdk/morphline/api/MorphlineCompilationException

    Details

    • Type: Improvement Improvement
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Won't Fix
    • Affects Version/s: v1.5.0
    • Fix Version/s: v1.6.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Test Flume+Solr, use flume 1.5 + solr 4.6.
      You can reproduce it by following steps
      1. Download http://www.apache.org/dyn/closer.cgi/flume/1.5.0/apache-flume-1.5.0-bin.tar.gz

      2. install: extract to /usr/lib/apache-flume-1.5.0-bin

      3. create /usr/lib/apache-flume-1.5.0-bin/conf/flume-conf-morphlineSolr.properties
      cat /usr/lib/apache-flume-1.5.0-bin/conf/flume-conf-morphlineSolr.properties
      a1.channels = c1
      a1.sources = r1
      a1.sinks = k1

      a1.channels.c1.type = memory

      a1.sources.r1.channels = c1
      a1.sources.r1.type = exec
      a1.sources.r1.command = tail -F /var/log/a1.new.log
      a1.sinks.k1.channel = c1

      a1.sinks.k1.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
      a1.sinks.k1.morphlineFile = /usr/lib/apache-flume-1.5.0-bin/conf/morphline.conf

      a1.channels.MemChannel.type = memory
      a1.channels.MemChannel.capacity = 10000
      a1.channels.MemChannel.transactionCapacity = 100

      4. create /usr/lib/apache-flume-1.5.0-bin/conf/morphline.conf
      cat /usr/lib/apache-flume-1.5.0-bin/conf/morphline.conf
      morphlines : [
      {
      id : morphline1

      importCommands : ["com.cloudera.**", "org.apache.solr.**"]

      commands : [
      {
      readLine

      { charset : UTF-8 }

      }

      {
      grok {
      dictionaryFiles : [/etc/flume/conf/grok-dictionaries]
      expressions : {
      message : """%

      {TIMESTAMP_LOG:timestamp}

      %

      {LOGLEVEL:loglevel}

      %

      {DATA:classname}

      : %

      {GREEDYDATA:msg}

      """

      }
      }
      }

      {
      convertTimestamp

      { field : timestamp inputFormats : ["yyyy-MM-dd'T'HH:mm:ss.SSS'Z'", "yyyy-MM-dd HH:mm:ss,SSS"] inputTimezone : America/Los_Angeles outputFormat : "yyyy-MM-dd'T'HH:mm:ss.SSS'Z'" outputTimezone : UTC }

      }

      {
      generateUUID

      { field : id }

      }
      {
      sanitizeUnknownSolrFields {
      solrLocator : {
      }
      }
      }

      { logInfo { format : "output record: {}", args : ["@{}"] } }

      {
      loadSolr {
      solrLocator : {
      }
      }
      }
      ]
      }
      ]

      5. start flume:
      cd /usr/lib/apache-flume-1.5.0-bin
      ./bin/flume-ng agent --conf conf --conf-file conf/flume-conf-morphlineSolr.properties --name a1
      folollowing error message found in log:

      29 May 2014 16:09:18,652 ERROR [lifecycleSupervisor-1-1] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@c0880a8 counterGroup:{ name:null counters:{} } } - Exception follows.
      java.lang.NoClassDefFoundError: org/kitesdk/morphline/api/MorphlineCompilationException
      at java.lang.Class.forName0(Native Method)
      at java.lang.Class.forName(Class.java:190)
      at org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:93)
      at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
      at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
      at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
      at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
      at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
      at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
      at java.lang.Thread.run(Thread.java:744)
      Caused by: java.lang.ClassNotFoundException: org.kitesdk.morphline.api.MorphlineCompilationException
      at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
      at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
      at java.security.AccessController.doPrivileged(Native Method)
      at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
      at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
      at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
      ... 13 more

        Activity

        Hide
        wolfgang hoschek added a comment -

        The dependencies have been made “optional” in flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml via <optional>true</optional>, thus the dependencies don’t ship automatically with the build.

        Here is a flume-centric way of getting hold of all the jars and plugging them into flume: https://groups.google.com/a/cloudera.org/d/msg/cdk-dev/7T4pTebdWN4/sBHGkoS70LkJ

        Show
        wolfgang hoschek added a comment - The dependencies have been made “optional” in flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml via <optional>true</optional>, thus the dependencies don’t ship automatically with the build. Here is a flume-centric way of getting hold of all the jars and plugging them into flume: https://groups.google.com/a/cloudera.org/d/msg/cdk-dev/7T4pTebdWN4/sBHGkoS70LkJ

          People

          • Assignee:
            Unassigned
            Reporter:
            liyunzhang_intel
          • Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

            • Created:
              Updated:
              Resolved:

              Development