1. Flume
  2. FLUME-1992

ElasticSearch dependency is marked optional


    • Type: Improvement Improvement
    • Status: Closed
    • Priority: Minor Minor
    • Resolution: Fixed
    • Affects Version/s: v1.4.0
    • Fix Version/s: v1.4.0
    • Component/s: Sinks+Sources
    • Labels:
    • Release Note:
      The JAR and JVM dependencies of the ElasticSearchSink are now documented in the Flume User Guide.


      FLUME-1371 added an ElasticSearch module but the dependency added in it for org.elasticsearch classes is marked <optional>true</optional>.

      This means it doesn't get generated into the builds Flume would ship and the user guide will not work unless a user manually places their version of ElasticSearch jars on the classpath. They will get the error:

      java.lang.NoClassDefFoundError: org/elasticsearch/common/transport/TransportAddress
              at java.lang.Class.forName0(Native Method)
              at java.lang.Class.forName(Class.java:188)
              at org.apache.flume.sink.DefaultSinkFactory.getClass(DefaultSinkFactory.java:67)
              at org.apache.flume.sink.DefaultSinkFactory.create(DefaultSinkFactory.java:41)
              at org.apache.flume.node.AbstractConfigurationProvider.loadSinks(AbstractConfigurationProvider.java:415)
              at org.apache.flume.node.AbstractConfigurationProvider.getConfiguration(AbstractConfigurationProvider.java:103)
              at org.apache.flume.node.PollingPropertiesFileConfigurationProvider$FileWatcherRunnable.run(PollingPropertiesFileConfigurationProvider.java:140)
              at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
              at java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.java:351)
              at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:165)
              at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:267)
              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
              at java.lang.Thread.run(Thread.java:679)
      Caused by: java.lang.ClassNotFoundException: org.elasticsearch.common.transport.TransportAddress
              at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
              at java.security.AccessController.doPrivileged(Native Method)
              at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:321)
              at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:294)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:266)
              ... 15 more

      Two ways to address this; either we:

      1. Mark dependency as non-optional, supply a bundled elasticsearch jar (version decision could be a problem I imagine, if their releases break compatibility often)
      2. Update the user guide to instruct the user to place the jar onto flume's classpath in proper manner, before following the remaining of the guide.

      Problem of the ClassNotFound originally reported by user j0sh_X over FreeNode IRC.

        Issue Links


          Harsh J created issue -
          Edward Sargisson made changes -
          Field Original Value New Value
          Attachment flume-1992-1.4-1.patch [ 12579901 ]
          Edward Sargisson made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 12170 ]
          Edward Sargisson made changes -
          Link This issue relates to FLUME-2015 [ FLUME-2015 ]
          Edward Sargisson made changes -
          Status Open [ 1 ] Resolved [ 5 ]
          Release Note The JAR and JVM dependencies of the ElasticSearchSink are now documented in the Flume User Guide.
          Fix Version/s v1.4.0 [ 12323372 ]
          Resolution Fixed [ 1 ]
          Edward Sargisson made changes -
          Status Resolved [ 5 ] Closed [ 6 ]


            • Assignee:
              Harsh J
            • Votes:
              0 Vote for this issue
              4 Start watching this issue


              • Created: