Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3219

Taildir source: if file is renamed, it is consumed again

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.8.0
    • None
    • Sinks+Sources
    • None

    Description

      Current behavior of Taildir is such that if a file is renamed (eg log rotated) it is consumed again.

      https://github.com/apache/flume/blob/d1f24f56ce9714bb3e1edc671da290c75a17dead/flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java#L247

      Would not be better if the inode is followed, and if that inode has been consumed do not consume it again? With current implementation, once file is rotated, you get duplicates in the case you include in the path previous days's data (you want to do that if agent fails and needs to consume data from previous days).

       

      Attachments

        Issue Links

          Blocked

          Bug - A problem which impairs or prevents the functions of the product. FLUME-3341 Taildir source may cause file handle leak and data duplication

          • Major - Major loss of function.
          • Patch Available

          Activity

            People

              Unassigned Unassigned
              DLanza Daniel Lanza García
              Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated: