Details
-
Improvement
-
Status: Open
-
Minor
-
Resolution: Unresolved
-
3.4.0
-
None
-
None
Description
In the current Hadoop 3.4 version, the system relies on Log4j 1.x for logging purposes. This dependency limits the logging functionality, especially when it comes to rolling log files such as 'hdfs-audit.log'. Rolling log files is crucial for managing log size and ensuring logs are rotated out over time to prevent excessive disk space usage. However, the Log4j 1.x version integrated within Hadoop lacks the necessary capabilities to efficiently handle log rolling.
This library extends logging capabilities, including more flexible and configurable log rolling features. By deploying this library, we can enable advanced rolling strategies such as time-based rolling, size-based rolling, and compression of rolled logs, which are not supported by the default Log4j 1.x setup in Hadoop.
The integration of Apache Log4j Extras into Hadoop will significantly improve log management by allowing for more sophisticated and configurable log rotation policies. This enhancement is particularly important for maintaining system performance and reliability, as well as for compliance with log retention policies.
Although there are plans to upgrade to Log4j 2 in the forthcoming Hadoop 3.5 version, which will inherently solve these issues by providing enhanced logging features, there is an immediate need to enable advanced log rolling capabilities in the current and previous versions of Hadoop.
