[NIFI-8344] Allow TailFile to continue tailing a file for some time after it has been rolled over - ASF JIRA

Agile Board

Attach files

Attach Screenshot

Voters

Watch issue

Watchers

Create sub-task

Link

Clone

Update Comment Author

Replace String in Comment

Update Comment Visibility

Delete Comments

XML

Word

Printable

JSON

Details

Type: Improvement
Status: Resolved
Priority: Major
Resolution: Fixed
Affects Version/s: None
Fix Version/s: 1.14.0
Component/s: Extensions
Labels:
None

Description

TailFile makes the assumption that once a file has been rolled over, it will never be appended to. If the file's Last Modified timestamp changes, the processor assumes that it's a new file and imports the entire contents of the file again.

However, one practice that I've encountered is that users have a syslog server that rotates periodically. To rotate, they rename the existing file, and then restart the server. When that happens, the server will flush out any data that it has buffered to the file that was just rolled over, and then begin writing to the new file.

This results in the TailFile processor ingesting the entire file that has been rolled over. Because we can't keep state about every file that is rolled over, we should introduce a property that allows the user to indicate that upon rollover they want to continue tailing that rolled over file until it is no longer being written to, and then begin tailing the new file.