Flume
  1. Flume
  2. FLUME-2340

Refactor to make room for Morphlines Elasticsearch Sink

    Details

    • Type: Improvement Improvement
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: v1.7.0
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Right now there are some non-Solr-specific classes in org.apache.flume.sink.solr.morphline and everything assumes data will get loaded into Solr. This should be refactored to make it possible to use Morphlines and send data to Elasticsearch, too, for example.

      See http://search-hadoop.com/m/Jrb3G1tSCQK1&subj=Re+Questions+about+Morphline+Solr+Sink+structure

        Activity

        Hide
        Otis Gospodnetic added a comment -

        Update for anyone looking to add Morphlines support for Elasticsearch to Flume: kite-morphlines-elasticsearch lives at https://github.com/sematext/kite-morphlines-elasticsearch

        Show
        Otis Gospodnetic added a comment - Update for anyone looking to add Morphlines support for Elasticsearch to Flume: kite-morphlines-elasticsearch lives at https://github.com/sematext/kite-morphlines-elasticsearch
        Show
        Pawel Rog added a comment - https://github.com/kite-sdk/kite/pull/47
        Hide
        wolfgang hoschek added a comment -

        Otis Gospodnetic Once someone put together the above parts we could take the next step - refactoring to have three flume maven modules: one module as a base, another module for solr dependencies, and yet another module for ES dependencies:

        • Add a new flume module flume-ng-morphline-sink that's a copy n paste of flume-ng-morphline-solr-sink except that it has a minimum of dependencies: depends on kite-morphlines-core instead of kite-morphlines-all, and in particular doesn't contain tests and classes that depend on solr (or ES). E.g. it contains MorphlineSink.java and MorphlineInterceptor.java but not MorphlineSolrSink.java.
        • Add flume-ng-morphline-elasticsearch-sink that depends on flume-ng-morphline-sink and also kite-morphlines-all-except-solr, plus kite-morphlines-elasticsearch. This means you can use class MorphlineSink from flume-ng-morphline-sink as is for ES.
        • Change flume-ng-morphline-solr-sink to depend on flume-ng-morphline-sink and also kite-morphlines-all. Retain backwards compat by retaining existing classes in flume-ng-morphline-solr-sink, yet pointing them to extend or use relevant things in flume-ng-morphline-sink
        Show
        wolfgang hoschek added a comment - Otis Gospodnetic Once someone put together the above parts we could take the next step - refactoring to have three flume maven modules: one module as a base, another module for solr dependencies, and yet another module for ES dependencies: Add a new flume module flume-ng-morphline-sink that's a copy n paste of flume-ng-morphline-solr-sink except that it has a minimum of dependencies: depends on kite-morphlines-core instead of kite-morphlines-all, and in particular doesn't contain tests and classes that depend on solr (or ES). E.g. it contains MorphlineSink.java and MorphlineInterceptor.java but not MorphlineSolrSink.java. Add flume-ng-morphline-elasticsearch-sink that depends on flume-ng-morphline-sink and also kite-morphlines-all-except-solr, plus kite-morphlines-elasticsearch. This means you can use class MorphlineSink from flume-ng-morphline-sink as is for ES. Change flume-ng-morphline-solr-sink to depend on flume-ng-morphline-sink and also kite-morphlines-all. Retain backwards compat by retaining existing classes in flume-ng-morphline-solr-sink, yet pointing them to extend or use relevant things in flume-ng-morphline-sink
        Hide
        wolfgang hoschek added a comment -

        Here is what enabling ES would involve:

        • Add a loadElasticSearch command in a corresponding kite-morphlines-elasticsearch maven module
        • change kite-morphlines-all to kite-morphlines-all-except-solr in flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml
        • add any necessary ES dependency jars to pom.xml
        • add unit tests and integration tests

        This doesn't require yet another sink - ES can be enabled with these packaging changes.

        Show
        wolfgang hoschek added a comment - Here is what enabling ES would involve: Add a loadElasticSearch command in a corresponding kite-morphlines-elasticsearch maven module change kite-morphlines-all to kite-morphlines-all-except-solr in flume-ng-sinks/flume-ng-morphline-solr-sink/pom.xml add any necessary ES dependency jars to pom.xml add unit tests and integration tests This doesn't require yet another sink - ES can be enabled with these packaging changes.

          People

          • Assignee:
            Unassigned
            Reporter:
            Otis Gospodnetic
          • Votes:
            1 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

            • Created:
              Updated:

              Development