Flume
  1. Flume
  2. FLUME-1992

ElasticSearch dependency is marked optional

    Details

    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.4.0
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Release Note:
      The JAR and JVM dependencies of the ElasticSearchSink are now documented in the Flume User Guide.

      Description

      FLUME-1371 added an ElasticSearch module but the dependency added in it for org.elasticsearch classes is marked <optional>true</optional>.

      This means it doesn't get generated into the builds Flume would ship and the user guide will not work unless a user manually places their version of ElasticSearch jars on the classpath. They will get the error:

      java.lang.NoClassDefFoundError: org/elasticsearch/common/transport/TransportAddress
              at java.lang.Class.forName0(Native Method)
              at java.lang.Class.forName(Class.java:188)
              at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
              at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
              at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
              at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
              at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:679)
      Caused by: java.lang.ClassNotFoundException: org.elasticsearch.common.transport.TransportAddress
              at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
              ... 15 more
      

      Two ways to address this; either we:

      1. Mark dependency as non-optional, supply a bundled elasticsearch jar (version decision could be a problem I imagine, if their releases break compatibility often)
      2. Update the user guide to instruct the user to place the jar onto flume's classpath in proper manner, before following the remaining of the guide.

      Problem of the ClassNotFound originally reported by user j0sh_X over FreeNode IRC.

        Issue Links

          Activity

          Hide
          Edward Sargisson added a comment -

          Documentation fix for this issue committed under FLUME-2015 as 2b66fb8d33d86bd7dabbb1b4f911c38c892dbdaa.

          Show
          Edward Sargisson added a comment - Documentation fix for this issue committed under FLUME-2015 as 2b66fb8d33d86bd7dabbb1b4f911c38c892dbdaa.
          Hide
          Edward Sargisson added a comment -

          Hi Mike,

          >Thanks for all your work on the ES sink Edward!
          Thanks.

          The doc patch on FLUME-2015 includes what is needed to fix this work item. I've left all the work items linked together until that patch goes in so that I can check everything over.

          Show
          Edward Sargisson added a comment - Hi Mike, >Thanks for all your work on the ES sink Edward! Thanks. The doc patch on FLUME-2015 includes what is needed to fix this work item. I've left all the work items linked together until that patch goes in so that I can check everything over.
          Hide
          Mike Percy added a comment -

          Happy to commit this unless this is obsoleted by FLUME-2015 in which case we should mark it a dup

          Thanks for all your work on the ES sink Edward!

          Mike

          Show
          Mike Percy added a comment - Happy to commit this unless this is obsoleted by FLUME-2015 in which case we should mark it a dup Thanks for all your work on the ES sink Edward! Mike
          Hide
          Edward Sargisson added a comment -

          Patch to update the documentation now included in FLUME-2015.

          Show
          Edward Sargisson added a comment - Patch to update the documentation now included in FLUME-2015 .
          Hide
          Edward Sargisson added a comment -

          User Guide patch attached

          Show
          Edward Sargisson added a comment - User Guide patch attached
          Hide
          Brock Noland added a comment -

          +1 for option #2, updating the user guide.

          Show
          Brock Noland added a comment - +1 for option #2, updating the user guide.
          Hide
          Harsh J added a comment -

          Thanks Edward. In j0sh's case, we needed the base elasticsearch "main" jar (pardon, I've never used ES) and a bunch of lucene dependencies it wanted in turn. We pulled them via wget/etc. from maven repo URLs but I think its saner to lift a bundle from an installed ES package location or so (like ES_HOME/lib/*.jar on classpath maybe).

          Show
          Harsh J added a comment - Thanks Edward. In j0sh's case, we needed the base elasticsearch "main" jar (pardon, I've never used ES) and a bunch of lucene dependencies it wanted in turn. We pulled them via wget/etc. from maven repo URLs but I think its saner to lift a bundle from an installed ES package location or so (like ES_HOME/lib/*.jar on classpath maybe).
          Hide
          Edward Sargisson added a comment -

          As indicated in the description, when elasticsearch makes a major release it ends to break compatibility with previously installed clusters.

          I know that 0.19 and 0.20 could not talk together. I have just tested having a 0.90RC2 client in the elasticsearchsink talking to a 0.20 cluster and it fails too.

          Thus I think that users should choose which elasticsearch jar they want to use and place it in the lib directory. Therefore option 2 above should be taken and the documentation updated.

          Show
          Edward Sargisson added a comment - As indicated in the description, when elasticsearch makes a major release it ends to break compatibility with previously installed clusters. I know that 0.19 and 0.20 could not talk together. I have just tested having a 0.90RC2 client in the elasticsearchsink talking to a 0.20 cluster and it fails too. Thus I think that users should choose which elasticsearch jar they want to use and place it in the lib directory. Therefore option 2 above should be taken and the documentation updated.

            People

            • Assignee:
              Unassigned
              Reporter:
              Harsh J
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development