[HIVE-24179] Memory leak in HS2 DbTxnManager when compiling SHOW LOCKS statement

• Type: Bug
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: 4.0.0-alpha-1
• Component/s: HiveServer2
• Labels: pull-request-available
• Assignee: Stamatis Zampetakis
• Reporter: Stamatis Zampetakis
• Created: 18/Sep/20 15:52
• Updated: 27/Feb/24 22:23
• Resolved: 13/Nov/20 17:25

Description

The problem can be reproduced by executing repeatedly a SHOW LOCK statement and monitoring the heap memory of HS2. For a small heap (e.g., 2g) it only takes a few minutes before the server crashes with OutOfMemory error such as the one shown below.

java.lang.OutOfMemoryError: GC overhead limit exceeded
        at java.util.Arrays.copyOf(Arrays.java:3332)
        at java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:124)
        at java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:448)
        at java.lang.StringBuilder.append(StringBuilder.java:136)
        at org.apache.maven.surefire.booter.ForkedChannelEncoder.encodeMessage(ForkedChannelEncoder.j
        at org.apache.maven.surefire.booter.ForkedChannelEncoder.setOutErr(ForkedChannelEncoder.java:
        at org.apache.maven.surefire.booter.ForkedChannelEncoder.stdErr(ForkedChannelEncoder.java:166
        at org.apache.maven.surefire.booter.ForkingRunListener.writeTestOutput(ForkingRunListener.jav
        at org.apache.maven.surefire.report.ConsoleOutputCapture$ForwardingPrintStream.write(ConsoleO
        at org.apache.logging.log4j.core.util.CloseShieldOutputStream.write(CloseShieldOutputStream.j
        at org.apache.logging.log4j.core.appender.OutputStreamManager.writeToDestination(OutputStream
        at org.apache.logging.log4j.core.appender.OutputStreamManager.flushBuffer(OutputStreamManager
        at org.apache.logging.log4j.core.appender.OutputStreamManager.flush(OutputStreamManager.java:
        at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(Abst
        at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutp
        at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputS
        at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:
        at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:12
        at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(Appender
        at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:84)
        at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:543)
        at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
        at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:485)
        at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:460)
        at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletio
        at org.apache.logging.log4j.core.Logger.log(Logger.java:162)
        at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2190)
        at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2
        at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2127)
        at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2008)
        at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1867)
        at org.apache.logging.slf4j.Log4jLogger.info(Log4jLogger.java:179)

The heap dump shows (summary.png) that most of the memory is consumed by Hashtable$Entry and ConcurrentHashMap$Node objects coming from Hive configurations referenced by DbTxnManager.

The latter are not eligible for garbage collection since at construction time they are passed implicitly in a callback stored inside ShutdownHookManager.

When the DbTxnManager is closed properly the leak is not present since the callback is removed from ShutdownHookManager.

SHOW LOCKS statements create (ShowDbLocksAnalyzer, ShowLocksAnalyzer) a new TxnManager and they never close it leading to the memory leak.

Issue Links

• links to: GitHub Pull Request #1509