[FLUME-1371] ElasticSearch Sink

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 1.3.0
• Fix Version/s: 1.3.0
• Component/s: Sinks+Sources
• Labels: noob
• Assignee: Cameron Gandevia
• Reporter: Mubarak Seyed
• Created: 15/Jul/12 05:36
• Updated: 08/Oct/14 20:39
• Resolved: 26/Oct/12 18:14

Description

Motivation:
Some production usecase(s) requires event(log) to be indexed and searched. There are workarounds (e.g. store the log in HDFS and run MapReduce job to create an index) but lack of near real-time indexing and search latency are concerns for searching the log records.

Design:
ElasticSearch supports RESTful style API to publish events for indexing.

There is an existing flumeelastic sink [1] in github but it does not support batching of events for indexing.

[1] https://github.com/Aconex/elasticflume

Issue Links

• links to: Review Board