[HIVE-22753] Fix gradual mem leak: Operationlog related appenders should be cleared up on errors

• Type: Improvement
• Status: Closed
• Priority: Minor
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: HiveServer2
• Assignee: Rajesh Balamohan
• Reporter: Rajesh Balamohan
• Created: 21/Jan/20 05:47
• Updated: 17/Nov/22 08:49
• Resolved: 28/Jan/20 03:12

Description

In case of exception in SQLOperation, operational log does not get cleared up. This causes gradual build up of HushableRandomAccessFileAppender causing HS2 to OOM after some time.

 

Allocation tree

 

Prod instance mem

 

Each HushableRandomAccessFileAppender holds internal ref to RandomAccessFileAppender which holds a 256 KB bytebuffer, causing the mem leak.

Related ticket: HIVE-18820

Issue Links

• causes: HIVE-25970 Missing messages in HS2 operation logs
• is a parent of: HIVE-24590 Operation Logging still leaks the log4j Appenders
• is related to: HIVE-17128 Operation Logging leaks file descriptors as the log4j Appender is never closed
• is related to: HIVE-24569 LLAP daemon leaks file descriptors/log4j appenders
• relates to: HIVE-22127 Query Routing logging appender is leaking resources of RandomAccessFileManager.