[FLUME-2446] Flume -> ElasticSearch loosing data

• Type: Bug
• Status: Open
• Priority: Major
• Resolution: Unresolved

Environment

Ubuntu 14.04.1 LTS

• Assignee: Unassigned
• Reporter: Tigran
• Created: 20/Aug/14 17:03
• Updated: 20/Aug/14 17:03

Description

I use flume to put events into elasticsearch, but not all my events are put there.
This is metrics json file:
{"SOURCE.avroSource":

{"OpenConnectionCount":"0","Type":"SOURCE","AppendBatchAcceptedCount":"0","AppendBatchReceivedCount":"0","EventAcceptedCount":"21","AppendReceivedCount":"21","StopTime":"0","StartTime":"1408553130069","EventReceivedCount":"21","AppendAcceptedCount":"21"}

,"SINK.esSyslogSink":

{"BatchCompleteCount":"0","ConnectionFailedCount":"0","EventDrainAttemptCount":"0","ConnectionCreatedCount":"1","Type":"SINK","BatchEmptyCount":"69","ConnectionClosedCount":"0","EventDrainSuccessCount":"0","StopTime":"0","StartTime":"1408553129240","BatchUnderflowCount":"0"}

,"CHANNEL.avroMemoryChannel":

{"EventPutSuccessCount":"21","ChannelFillPercentage":"0.0","Type":"CHANNEL","StopTime":"0","EventPutAttemptCount":"21","ChannelSize":"0","StartTime":"1408553129232","EventTakeSuccessCount":"21","ChannelCapacity":"1000","EventTakeAttemptCount":"162"}

,"SINK.esAvroSink":

{"BatchCompleteCount":"2","ConnectionFailedCount":"0","EventDrainAttemptCount":"14","ConnectionCreatedCount":"1","Type":"SINK","BatchEmptyCount":"68","ConnectionClosedCount":"0","EventDrainSuccessCount":"14","StopTime":"0","StartTime":"1408553129245","BatchUnderflowCount":"1"}

,"CHANNEL.syslogMemoryChannel":{"EventPutSuccessCount":"0","ChannelFillPercentage":"0.0","Type":"CHANNEL","StopTime":"0","EventPutAttemptCount":"0","ChannelSize":"0","StartTime":"1408553129232","EventTakeSuccessCount":"0","ChannelCapacity":"1000","EventTakeAttemptCount":"69"}}

So I put 21 events, and only 14 were passed into elasticsearch (EventDrainSuccessCount":"14"). Flume do receive all 21 events: "EventAcceptedCount":"21", and also my written interceptor (that log every incoming message) shows me this. Ration sent/received is random: sometime it gets all messages, but usually it is between 6-14 of 21.

Do somebody experienced this? No errors or warnings in the log.