[FLUME-2390] Flume-ElasticSearch Data gets posted multiple times when one of the event fail validation at elastic search sink for JSON Data

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Won't Fix
• Affects Version/s: 1.4.0
• Component/s: Sinks+Sources

Environment

CDH4.5

• Assignee: Unassigned
• Reporter: Deepak Subhramanian
• Created: 22/May/14 11:18
• Updated: 08/Oct/22 17:41
• Resolved: 08/Oct/22 17:41

Description

Hi,

I am using Elastic Search Sink to post JSON data. I used the temporary fix mentioned in https://issues.apache.org/jira/browse/FLUME-2126 to get JSON data posted to elastic search. When one of the message fail validation at ElasticSearch mapping for JSON data ( For example - getting empty message) , Flume seems to post the entire batch again and again until I restart Flume. Because of that no of events went from an avg of 100 to avg of 2000 per 10 minutes. As a temporary fix I set a header in my FlumeHTTP Source for non valid JSON and used a interceptor to send data to multiple ESSINKS which has different index names.