Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
2.0.0, 1.28.1
-
None
-
Patch
Description
Just setting the KeepSourceFile property to true can cause continuous ingestion of files into NiFi. If the file is big (e.g. 20 GB) this can cause the content repository (e.g. size of 400 GB) to be filled in an instant. This renders the NiFi node unusable and a cleanup is needed. There is no reason for this to happen, the flow should at least have enough time to process a chunk of such a huge file before attempting to load the same file again.
A quick solution would be just to add
@DefaultSchedule(strategy = SchedulingStrategy.TIMER_DRIVEN, period = "1 min")
This is anyway present on the ListFile processor, so why not to add it here also? if the user really wants to set this to 0 seconds I guess he should be aware of the consequences.