Details
-
Bug
-
Status: Open
-
Major
-
Resolution: Unresolved
-
1.5.0
-
None
-
None
-
VMs using RedHat
Description
Hi there,
in my current work I have come to discover a behaviour of Flume that doesn't seem to be expected. Explanation:
1- Architecture (flume 1.5, not tested on 1.6)
a- flume server = virtual machine running one (and one only) flume server
b- First Flume server (receives 1 flow of data and transfers it using AVRO to another Flume Server.
c- the 2nd Flume replicates the source 4 times to 4 different sinks (the final sinks)
d- All flume servers use the file channel
2- Problem description:
a- one of the final sinks fails
b- the channel feeding it buffers and fills up
c- the first flume then starts buffering until channel is full even though the 3 other final sinks are still receiving and working correctly
d- the channel is full on flume 1 and starts complaining.
e- when the faulty sink is back again, flume 1 flushes its events
f- this causes the events to be written multiple times into the 3 other non faulty final sinks in addition to the faulty one => very very high number of duplication (up to 24)
I believe that this is a bug and I did not see it reported by someone else. Please let me know if I am mistaken.
thank you,
Adam.