[NIFI-8345] Allow TailFile to continue tailing a file for some time after it has been rolled over

• Type: Improvement
• Status: Resolved
• Priority: Major
• Resolution: Duplicate
• Component/s: Extensions
• Assignee: Mark Payne
• Reporter: Mark Payne
• Created: 18/Mar/21 21:19
• Updated: 18/Mar/21 21:20
• Resolved: 18/Mar/21 21:20

Description

TailFile makes the assumption that once a file has been rolled over, it will never be appended to. If the file's Last Modified timestamp changes, the processor assumes that it's a new file and imports the entire contents of the file again.

However, one practice that I've encountered is that users have a syslog server that rotates periodically. To rotate, they rename the existing file, and then restart the server. When that happens, the server will flush out any data that it has buffered to the file that was just rolled over, and then begin writing to the new file.

This results in the TailFile processor ingesting the entire file that has been rolled over. Because we can't keep state about every file that is rolled over, we should introduce a property that allows the user to indicate that upon rollover they want to continue tailing that rolled over file until it is no longer being written to, and then begin tailing the new file.

Issue Links

• duplicates: NIFI-8344 Allow TailFile to continue tailing a file for some time after it has been rolled over