Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Won't Fix
-
1.5.0
-
None
Description
Test Flume+Solr, use flume 1.5 + solr 4.6.
You can reproduce it by following steps
1. Download http://www.apache.org/dyn/closer.cgi/flume/1.5.0/apache-flume-1.5.0-bin.tar.gz
2. install: extract to /usr/lib/apache-flume-1.5.0-bin
3. create /usr/lib/apache-flume-1.5.0-bin/conf/flume-conf-morphlineSolr.properties
cat /usr/lib/apache-flume-1.5.0-bin/conf/flume-conf-morphlineSolr.properties
a1.channels = c1
a1.sources = r1
a1.sinks = k1
a1.channels.c1.type = memory
a1.sources.r1.channels = c1
a1.sources.r1.type = exec
a1.sources.r1.command = tail -F /var/log/a1.new.log
a1.sinks.k1.channel = c1
a1.sinks.k1.type = org.apache.flume.sink.solr.morphline.MorphlineSolrSink
a1.sinks.k1.morphlineFile = /usr/lib/apache-flume-1.5.0-bin/conf/morphline.conf
a1.channels.MemChannel.type = memory
a1.channels.MemChannel.capacity = 10000
a1.channels.MemChannel.transactionCapacity = 100
4. create /usr/lib/apache-flume-1.5.0-bin/conf/morphline.conf
cat /usr/lib/apache-flume-1.5.0-bin/conf/morphline.conf
morphlines : [
{
id : morphline1
importCommands : ["com.cloudera.**", "org.apache.solr.**"]
commands : [
{
readLine
}
{
grok {
dictionaryFiles : [/etc/flume/conf/grok-dictionaries]
expressions : {
message : """%
%
{LOGLEVEL:loglevel}%
{DATA:classname}: %
{GREEDYDATA:msg}"""
}
}
}
{
convertTimestamp
}
{
generateUUID
}
{
sanitizeUnknownSolrFields {
solrLocator : {
}
}
}
{ logInfo { format : "output record: {}", args : ["@{}"] } }
{
loadSolr {
solrLocator : {
}
}
}
]
}
]
5. start flume:
cd /usr/lib/apache-flume-1.5.0-bin
./bin/flume-ng agent --conf conf --conf-file conf/flume-conf-morphlineSolr.properties --name a1
folollowing error message found in log:
29 May 2014 16:09:18,652 ERROR [lifecycleSupervisor-1-1] (org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run:253) - Unable to start SinkRunner: { policy:org.apache.flume.sink.DefaultSinkProcessor@c0880a8 counterGroup:{ name:null counters:{} } } - Exception follows.
java.lang.NoClassDefFoundError: org/kitesdk/morphline/api/MorphlineCompilationException
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at org.apache.flume.sink.solr.morphline.MorphlineSink.start(MorphlineSink.java:93)
at org.apache.flume.sink.DefaultSinkProcessor.start(DefaultSinkProcessor.java:46)
at org.apache.flume.SinkRunner.start(SinkRunner.java:79)
at org.apache.flume.lifecycle.LifecycleSupervisor$MonitorRunnable.run(LifecycleSupervisor.java:251)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.ClassNotFoundException: org.kitesdk.morphline.api.MorphlineCompilationException
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
... 13 more