[HIVE-17128] Operation Logging leaks file descriptors as the log4j Appender is never closed

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 3.0.0
• Component/s: Logging
• Assignee: Andrew Sherman
• Reporter: Andrew Sherman
• Created: 19/Jul/17 19:54
• Updated: 06/Jan/21 10:36
• Resolved: 26/Jul/17 09:52

Description

HIVE-16061 and HIVE-16400 changed operation logging to use the Log4j2 RoutingAppender to automatically output the log for each query into each individual operation log file. As log4j does not know when a query is finished it keeps the OutputStream in the Appender open even when the query completes. The stream holds a file descriptor and so we leak file descriptors. Note that we are already careful to close any streams reading from the operation log file.

Fix

To fix this we use a technique described in the comments of LOG4J2-510 which uses reflection to close the appender. The test in TestOperationLoggingLayout will be extended to check that the Appender is closed.

Issue Links

• is related to: HIVE-24569 LLAP daemon leaks file descriptors/log4j appenders
• is related to: HIVE-24590 Operation Logging still leaks the log4j Appenders
• relates to: HIVE-22753 Fix gradual mem leak: Operationlog related appenders should be cleared up on errors