Details
-
Improvement
-
Status: Open
-
Major
-
Resolution: Unresolved
-
None
-
None
-
None
Description
The NiFi API and documentation have always been pretty clear that Processors are responsible for properly committing/rolling back their Process Sessions. However, what happens when a Processor fails to do so?
The AbstractProcessor (from which 99+% of all Processors extend) will automatically call ProcessSession.commit() if the subclass's onTrigger() method returns successfully and call ProcessSession.rollback() if any Throwable is thrown from the subclass's onTrigger().
However, those Processors that extend from AbstractSessionFactoryProcessor do not have this protection. As a fail-safe, we have always had a finalizer in StandardProcessSession. The finalizer simply calls rollback(). This ensures that any FlowFiles that were not properly handled (transferred, removed, etc.) are accounted for and re-queued/discarded.
Java 9 has now made finalizers deprecated, though. Additionally, through doing some research, I have seen that the finalizer ends up being quite expensive in the effects that it causes on the system's Garbage Collection. Specifically, GC ends up reclaiming less and less and less heap until a point at which a lot more is reclaimed. This results in a JVM that is sometimes "choppy" in its responsiveness, which can lead to instabilities. (Will attach screenshots showing this behavior).
Given this, and given that only a handful of Processors need to make use of the AbstractSessionFactoryProcessor, we should remove the Process Session's finalizer.
