[FLUME-3254] taildir source works abnormally when the monitored file has too fast append speed．

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Not A Problem
• Affects Version/s: 1.8.0
• Fix Version/s: notrack
• Component/s: Sinks+Sources

Environment

flume 1.8.0 virtual-machine-ubuntu16.04 desktop

• Assignee: Unassigned
• Reporter: Alan Liu
• Created: 09/Jul/18 06:37
• Updated: 18/Jul/18 07:26
• Resolved: 18/Jul/18 07:26

Description

Hello,

I am using flume 1.8.0 for ubuntu 16.04.  I met an question when using  the agent, taildir-memory-fileroll. I programed for appending  data into the file monitored by taildir every 2 second．Then at this speed the agent worked abnormally. Although the json file generated by taildir always updated and recorded the newest position, the file generated by rollfile didn't have any more data until I killed the bin which is appending data into the file monitored by taildir. When I changed the file appending interval to be 3 second. Everything worked normally. the following is my agent configuration file.

#a1 taildir-----> fileroll
a1.sources = src1
a1.sinks = sink1
a1.channels = chan1

a1.sources.src1.type = TAILDIR
a1.sources.src1.positionFile = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/position.log
a1.sources.src1.filegroups =f1
a1.sources.src1.filegroups.f1 = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/log2.txt
a1.sources.src1.writePosInterval=500

a1.channels.chan1.type=memory
a1.channels.chan1.capacity=100000
a1.channels.chan1.transactionCapacity=10000

a1.sinks.sink1.channel = chan1
a1.sinks.sink1.type = file_roll
a1.sinks.sink1.sink.directory = /home/EFM/Downloads/apache-flume-1.8.0-bin/FileRoll
a1.sinks.sink1.sink.rollInterval=0

a1.sources.src1.channels = chan1
a1.sinks.sink1.channel = chan1

Would you like give me some suggestions for that questions?

thanks.