Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3254

taildir source works abnormally when the monitored file has too fast append speed.

    Details

    • Type: Bug
    • Status: Resolved
    • Priority: Major
    • Resolution: Not A Problem
    • Affects Version/s: 1.8.0
    • Fix Version/s: notrack
    • Component/s: Sinks+Sources
    • Labels:
      None
    • Environment:

      flume 1.8.0 virtual-machine-ubuntu16.04 desktop

      Description

      Hello,

      I am using flume 1.8.0 for ubuntu 16.04.  I met an question when using  the agent, taildir-memory-fileroll. I programed for appending  data into the file monitored by taildir every 2 second.Then at this speed the agent worked abnormally. Although the json file generated by taildir always updated and recorded the newest position, the file generated by rollfile didn't have any more data until I killed the bin which is appending data into the file monitored by taildir. When I changed the file appending interval to be 3 second. Everything worked normally. the following is my agent configuration file.

      #a1 taildir-----> fileroll
      a1.sources = src1
      a1.sinks = sink1
      a1.channels = chan1

      a1.sources.src1.type = TAILDIR
      a1.sources.src1.positionFile = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/position.log
      a1.sources.src1.filegroups =f1
      a1.sources.src1.filegroups.f1 = /home/EFM/Downloads/apache-flume-1.8.0-bin/TailDir/log2.txt
      a1.sources.src1.writePosInterval=500

      a1.channels.chan1.type=memory
      a1.channels.chan1.capacity=100000
      a1.channels.chan1.transactionCapacity=10000

      a1.sinks.sink1.channel = chan1
      a1.sinks.sink1.type = file_roll
      a1.sinks.sink1.sink.directory = /home/EFM/Downloads/apache-flume-1.8.0-bin/FileRoll
      a1.sinks.sink1.sink.rollInterval=0

      a1.sources.src1.channels = chan1
      a1.sinks.sink1.channel = chan1

      Would you like give me some suggestions for that questions?

      thanks.

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              AlanLiu Alan Liu
            • Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: