Details
-
Improvement
-
Status: Closed
-
Minor
-
Resolution: Fixed
-
1.4.0
-
None
-
The JAR and JVM dependencies of the ElasticSearchSink are now documented in the Flume User Guide.
Description
FLUME-1371 added an ElasticSearch module but the dependency added in it for org.elasticsearch classes is marked <optional>true</optional>.
This means it doesn't get generated into the builds Flume would ship and the user guide will not work unless a user manually places their version of ElasticSearch jars on the classpath. They will get the error:
java.lang.NoClassDefFoundError: org/elasticsearch/common/transport/TransportAddress at java.lang.Class.forName0(Native Method) at java.lang.Class.forName(Class.java:188) at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67) at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41) at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415) at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103) at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351) at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:679) Caused by: java.lang.ClassNotFoundException: org.elasticsearch.common.transport.TransportAddress at java.net.URLClassLoader$1.run(URLClassLoader.java:217) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:205) at java.lang.ClassLoader.loadClass(ClassLoader.java:321) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294) at java.lang.ClassLoader.loadClass(ClassLoader.java:266) ... 15 more
Two ways to address this; either we:
- Mark dependency as non-optional, supply a bundled elasticsearch jar (version decision could be a problem I imagine, if their releases break compatibility often)
- Update the user guide to instruct the user to place the jar onto flume's classpath in proper manner, before following the remaining of the guide.
Problem of the ClassNotFound originally reported by user j0sh_X over FreeNode IRC.
Attachments
Attachments
Issue Links
- relates to
-
FLUME-2015 ElasticSearchSink: need access to IndexRequestBuilder instance during flume event processing
- Resolved
- links to
As indicated in the description, when elasticsearch makes a major release it ends to break compatibility with previously installed clusters.
I know that 0.19 and 0.20 could not talk together. I have just tested having a 0.90RC2 client in the elasticsearchsink talking to a 0.20 cluster and it fails too.
Thus I think that users should choose which elasticsearch jar they want to use and place it in the lib directory. Therefore option 2 above should be taken and the documentation updated.