Uploaded image for project: 'Flume'
  1. Flume
  2. FLUME-3219

Taildir source: if file is renamed, it is consumed again

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: 1.8.0
    • Fix Version/s: None
    • Component/s: Sinks+Sources
    • Labels:
      None

      Description

      Current behavior of Taildir is such that if a file is renamed (eg log rotated) it is consumed again.

      https://github.com/apache/flume/blob/d1f24f56ce9714bb3e1edc671da290c75a17dead/flume-ng-sources/flume-taildir-source/src/main/java/org/apache/flume/source/taildir/ReliableTaildirEventReader.java#L247

      Would not be better if the inode is followed, and if that inode has been consumed do not consume it again? With current implementation, once file is rotated, you get duplicates in the case you include in the path previous days's data (you want to do that if agent fails and needs to consume data from previous days).

       

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              DLanza Daniel Lanza García
            • Votes:
              1 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated: