Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
1.16.3
-
None
-
All
Description
Problem statement and context
NiFi intentionally captures and redirects everything sent to standard out and standard error. It does so in the RunNiFi class, seen here.
This approach has the benefit of assigning logback logger classification to output that is sent to standard out and standard error. Thus, messages that are printed to standard out are labeled as originating from the "org.apache.nifi.StdOut" logback logger, and output from the "NiFi logging handler" thread. This catches all stray messages that may casually be output to standard out or standard error for debugging purposes...
This approach also has drawbacks. It does not permit the NiFi administrator to redirect standard error or standard out easily. For example, if NiFi is configured to write all logs to standard out, this "encases" every log message as if it were written to the "org.apache.nifi.StdOut" logger at the INFO log level. For example, suppose logback is configured to write all messages to the console (standard out) with the message format: [date] [log level] [thread] [logger] [message]. All subsequent output would be "double wrapped", first being output by the logger it was emitted from, then being output by the logger handling standard out, and would look like the sample below. (Red is the "wrapping" logger, blue is the original message).
[date] INFO [NiFi logging handler] org.apache.nifi.StdOut [date] [original log level] [original thread] [log message]
This approach has an additional drawback that all log messages emitted from org.apache.nifi.StdOut are output at the INFO log level. Furthermore, all log messages emitted from org.apache.nifi.StdErr are emitted at the ERROR log level. This secludes the log level from the source logger, making log filtering more difficult if the log messages passes through standard out or standard error. Every log message is forcibly set to one log level.
Why this should be improved
It would be beneficial for system administrators to allow redirection of standard out and standard error. The ability to pipe log messages to the external logging solution of their choice would be an important improvement. For example, when NiFi is run in a Docker container, Docker can send logs elsewhere using its configurable log driver. This means Docker is responsible for buffering the logs and forwarding them on to an external log solution such as syslog, AWS CloudWatch, Fluent, Google Cloud Logging, Splunk, and others.
It is beneficial to use an external log solution so that logs do not fill up the disk of the computer running NiFi.
Proposed improvement
There are at least two approaches that would improve NiFi.
Approach #1: remove the redirection of standard out and standard error in the RunNiFi class. This would be the most disruptive approach from a deployment perspective, and may affect some existing instances of NiFi. This is the easiest technical solution and the most simple.
Approach #2: add a configuration parameter to bootstrap.conf to disable redirection of standard out and standard error. The default would be that redirection does still occur. This would be the least disruptive approach, and would not affect existing instances of NiFi unless the system administrator specifically configures the setting.
Questions and Implementation
- Would the NiFi developer community be amenable to this change?
- If so, which approach is preferred?
If there is a positive response to this improvement, I will follow the Contributor Guide and implement the desired improvement approach, then submit a pull request.
Thank you for your time and attention.