Details

    • Type: New Feature New Feature
    • Status: Resolved
    • Priority: Major Major
    • Resolution: Fixed
    • Affects Version/s: v1.3.0
    • Fix Version/s: v1.3.0
    • Component/s: Sinks+Sources
    • Labels:

      Description

      Motivation:
      Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.

      Design:
      ElasticSearch supports RESTful style API to publish events for indexing.

      There is an existing flumeelastic sink [1] in github but it does not support batching of events for indexing.

      [1] https://github.com/Aconex/elasticflume

      1. FLUME-1371.patch
        61 kB
        Cameron Gandevia

        Issue Links

          Activity

          Mubarak Seyed created issue -
          Hide
          Cameron Gandevia added a comment -

          First cut of an elasticsearch sink I plan on using. It is based off of the existing flumeelastic sink

          Show
          Cameron Gandevia added a comment - First cut of an elasticsearch sink I plan on using. It is based off of the existing flumeelastic sink
          Cameron Gandevia made changes -
          Field Original Value New Value
          Attachment FLUME-1371.patch [ 12548984 ]
          Hide
          Otis Gospodnetic added a comment -

          +1!
          Should Fix Version be set to 1.3.0 so this gets included in the 1.3.0 release?

          Cameron Gandevia Thanks! Are you actually using this somewhere and can you comments on the stability, performance, etc.?

          Show
          Otis Gospodnetic added a comment - +1! Should Fix Version be set to 1.3.0 so this gets included in the 1.3.0 release? Cameron Gandevia Thanks! Are you actually using this somewhere and can you comments on the stability, performance, etc.?
          Cameron Gandevia made changes -
          Status Open [ 1 ] Patch Available [ 10002 ]
          Affects Version/s v1.3.0 [ 12322140 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12549564 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12548984 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12549567 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12549567 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12549569 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12549564 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12550111 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12549569 ]
          Hide
          Jorn Argelo added a comment -

          As a humble user I'd like to say that it would be great to see this going into 1.3.0. That means I no longer need a rabbitmq or something like that to put stuff into Elastic Search.

          Show
          Jorn Argelo added a comment - As a humble user I'd like to say that it would be great to see this going into 1.3.0. That means I no longer need a rabbitmq or something like that to put stuff into Elastic Search.
          Hide
          Hari Shreedharan added a comment -

          Tentatively setting Fix Version as 1.3.

          Brock Noland - Please change if you feel this may not make it.

          Show
          Hari Shreedharan added a comment - Tentatively setting Fix Version as 1.3. Brock Noland - Please change if you feel this may not make it.
          Hari Shreedharan made changes -
          Fix Version/s v1.3.0 [ 12322140 ]
          Hide
          Hari Shreedharan added a comment -

          Brock Noland - You might need to make some manual changes to poms to make sure it commits to all branches without issue.

          Show
          Hari Shreedharan added a comment - Brock Noland - You might need to make some manual changes to poms to make sure it commits to all branches without issue.
          Hide
          Cameron Gandevia added a comment -

          Fixed EventSerializer to push through Malformed JSON as text

          Show
          Cameron Gandevia added a comment - Fixed EventSerializer to push through Malformed JSON as text
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12550679 ]
          Hide
          Brock Noland added a comment -

          1.3 sounds good for this.

          Show
          Brock Noland added a comment - 1.3 sounds good for this.
          Brock Noland made changes -
          Remote Link This issue links to "Review Board (Web Link)" [ 11404 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12550852 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12550111 ]
          Cameron Gandevia made changes -
          Attachment FLUME-1371.patch [ 12550679 ]
          Brock Noland made changes -
          Assignee Cameron Gandevia [ gnoremac ]
          Hide
          Brock Noland added a comment -

          Committed to trunk, 1.4, and 1.3. Thank you for the contribution Cameron!

          Show
          Brock Noland added a comment - Committed to trunk, 1.4, and 1.3. Thank you for the contribution Cameron!
          Brock Noland made changes -
          Status Patch Available [ 10002 ] Resolved [ 5 ]
          Resolution Fixed [ 1 ]
          Hide
          Hudson added a comment -

          Integrated in flume-trunk #319 (See https://builds.apache.org/job/flume-trunk/319/)
          FLUME-1371: ElasticSearch Sink (Revision 6d083df627e7513307d328dd9c92748bd055de0b)

          Result = SUCCESS
          brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6d083df627e7513307d328dd9c92748bd055de0b
          Files :

          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkTest.java
          • flume-ng-sinks/pom.xml
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchDynamicSerializerTest.java
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkConstants.java
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializerTest.java
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java
          • flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml
          • pom.xml
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchDynamicSerializer.java
          • flume-ng-dist/pom.xml
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java
          • flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchEventSerializer.java
          • flume-ng-doc/sphinx/FlumeUserGuide.rst
          Show
          Hudson added a comment - Integrated in flume-trunk #319 (See https://builds.apache.org/job/flume-trunk/319/ ) FLUME-1371 : ElasticSearch Sink (Revision 6d083df627e7513307d328dd9c92748bd055de0b) Result = SUCCESS brock : http://git-wip-us.apache.org/repos/asf/flume/repo?p=flume.git&a=commit&h=6d083df627e7513307d328dd9c92748bd055de0b Files : flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkTest.java flume-ng-sinks/pom.xml flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchDynamicSerializerTest.java flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSink.java flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchSinkConstants.java flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializerTest.java flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ContentBuilderUtil.java flume-ng-sinks/flume-ng-elasticsearch-sink/pom.xml pom.xml flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchLogStashEventSerializer.java flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchDynamicSerializer.java flume-ng-dist/pom.xml flume-ng-sinks/flume-ng-elasticsearch-sink/src/test/java/org/apache/flume/sink/elasticsearch/AbstractElasticSearchSinkTest.java flume-ng-sinks/flume-ng-elasticsearch-sink/src/main/java/org/apache/flume/sink/elasticsearch/ElasticSearchEventSerializer.java flume-ng-doc/sphinx/FlumeUserGuide.rst

            People

            • Assignee:
              Cameron Gandevia
              Reporter:
              Mubarak Seyed
            • Votes:
              1 Vote for this issue
              Watchers:
              10 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Development