Flume
  1. Flume
  2. FLUME-2390

Flume-ElasticSearch Data gets posted multiple times when one of the event fail validation at elastic search sink for JSON Data

    Details

    • Type: Bug Bug
    • Status: Open
    • Priority: Major Major
    • Resolution: Unresolved
    • Affects Version/s: v1.4.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      CDH4.5

      Description

      Hi,

      I am using Elastic Search Sink to post JSON data. I used the temporary fix mentioned in https://issues.apache.org/jira/browse/FLUME-2126 to get JSON data posted to elastic search. When one of the message fail validation at ElasticSearch mapping for JSON data ( For example - getting empty message) , Flume seems to post the entire batch again and again until I restart Flume. Because of that no of events went from an avg of 100 to avg of 2000 per 10 minutes. As a temporary fix I set a header in my FlumeHTTP Source for non valid JSON and used a interceptor to send data to multiple ESSINKS which has different index names.

        Activity

        No work has yet been logged on this issue.

          People

          • Assignee:
            Unassigned
            Reporter:
            Deepak Subhramanian
          • Votes:
            0 Vote for this issue
            Watchers:
            8 Start watching this issue

            Dates

            • Created:
              Updated:

              Development