[FLUME-2099] Add serializer to ElasticSearchSink for new logstash v1 format

• Type: Improvement
• Status: Open
• Priority: Major
• Resolution: Unresolved
• Affects Version/s: 1.4.0
• Component/s: Sinks+Sources
• Assignee: Unassigned
• Reporter: Edward Sargisson
• Created: 21/Jun/13 17:43
• Updated: 24/Oct/13 17:29

Description

The logstash maintainer has implemented an improved message schema for elasticsearch which he plans to deploy in the next version https://logstash.jira.com/browse/LOGSTASH-675

The ElasticSearchSink aims to allow users to write just as if logstash does - largely so Kibana can work. A serializer should be added to allow the new format to be used.

Benefits (from the logstash work item):

• kibana isn't polluted with "@" symbols everywhere
• most relevant data is in 'event fields' which is now top-level, no longer "@fields.somefield"
• fewer "required" event fields.
• the 'json' input format can go away or generally mean the same thing as json_event

Issue Links

• is depended upon by: FLUME-2220 ElasticSearch sink - duplicate fields in indexed document