Details
-
Bug
-
Status: Closed
-
Major
-
Resolution: Not A Bug
-
None
-
None
-
None
-
None
Description
My application is a sprint boot application that uses log4j2 and runs in a Wildfly server. After the zero day attack, we upgraded to the latest log4j2 version(2.16). But after the log4j upgrade, my application stops working once in a while. And when I looked at the threaddumps, I found that there is a deadlock created by log4j.
When analysing this issue, I came through a possible defect in log4j code. Not sure if that can result in a deadlock.
Log4J possible bug - As per the release notes, there was a fix to Enable immediate flush on RollingFileAppender when buffered i/o is not enabled. (LOG4J2-3114). But the code just does the opposite in RollingFileAppenderBuilder.
It should have been if(!bufferedIo) { immediateFlush = true; }{}. And one of my appender explicitly sets bufferedIo value to true. I know that log4j does a bufferedio by default and it is not necessary to set this flag explicitly. But unfortunately the code that I am working on is a legacy code and the configuration was working fine before the upgrade.
I have added my log4j configuration in here https://stackoverflow.com/questions/70450611/log4j2-deadlock
Extract from Thread dump:
"default task-128" #450 prio=5 os_prio=0 tid=0x00007f31f80cf800 nid=0x14c8 waiting for monitor entry [0x00007f31a7d88000]
java.lang.Thread.State: BLOCKED (on object monitor)
at org.apache.logging.log4j.core.appender.OutputStreamManager.writeBytes(OutputStreamManager.java:352)
- waiting to lock <0x00000000c0e70eb0> (a org.apache.logging.log4j.core.appender.OutputStreamManager)
at org.apache.logging.log4j.core.layout.TextEncoderHelper.writeEncodedText(TextEncoderHelper.java:96)
at org.apache.logging.log4j.core.layout.TextEncoderHelper.encodeText(TextEncoderHelper.java:65)
at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:68)
at org.apache.logging.log4j.core.layout.StringBuilderEncoder.encode(StringBuilderEncoder.java:32)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:228)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:60)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:197)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:190)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:181)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:542)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:500)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
at org.apache.logging.log4j.core.config.LoggerConfig.logParent(LoggerConfig.java:533)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:502)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:483)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:388)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:63)
at org.apache.logging.log4j.core.Logger.logMessage(Logger.java:153)
at org.apache.logging.slf4j.Log4jLogger.log(Log4jLogger.java:376)
at org.apache.commons.logging.impl.SLF4JLocationAwareLog.error(SLF4JLocationAwareLog.java:203)
at org.springframework.boot.web.support.ErrorPageFilter.handleCommittedResponse(ErrorPageFilter.java:225)