[SPARK-38446] Deadlock between ExecutorClassLoader and FileDownloadCallback caused by Log4j

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.0.3, 3.1.2, 3.2.1
• Fix Version/s: 3.1.3, 3.3.0, 3.2.2
• Component/s: Spark Core
• Assignee: Kent Yao 2
• Reporter: Kent Yao 2
• Created: 08/Mar/22 09:37
• Updated: 27/Apr/22 08:01
• Resolved: 03/Apr/22 17:31

Description

files-client-8-1
PRIORITY : 5

THREAD ID : 0X00007FBFFC5EE000

NATIVE ID : 0X14903

NATIVE ID (DECIMAL) : 84227

STATE : BLOCKED


stackTrace:
java.lang.Thread.State: BLOCKED (on object monitor)
at java.lang.ClassLoader.loadClass(ClassLoader.java:398)
- waiting to lock <0x00000003c0753f88> (a org.apache.spark.repl.ExecutorClassLoader)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
at org.apache.logging.log4j.util.LoaderUtil.loadClass(LoaderUtil.java:169)
at org.apache.logging.log4j.core.impl.ThrowableProxyHelper.loadClass(ThrowableProxyHelper.java:214)
at org.apache.logging.log4j.core.impl.ThrowableProxyHelper.toExtendedStackTrace(ThrowableProxyHelper.java:112)
at org.apache.logging.log4j.core.impl.ThrowableProxy.(ThrowableProxy.java:113)
at org.apache.logging.log4j.core.impl.ThrowableProxy.(ThrowableProxy.java:97)
at org.apache.logging.log4j.core.impl.Log4jLogEvent.getThrownProxy(Log4jLogEvent.java:629)
at org.apache.logging.log4j.core.pattern.ExtendedThrowablePatternConverter.format(ExtendedThrowablePatternConverter.java:63)
at org.apache.logging.log4j.core.layout.PatternLayout$NoFormatPatternSerializer.toSerializable(PatternLayout.java:342)
at org.apache.logging.log4j.core.layout.PatternLayout.toText(PatternLayout.java:240)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:225)
at org.apache.logging.log4j.core.layout.PatternLayout.encode(PatternLayout.java:59)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.directEncodeEvent(AbstractOutputStreamAppender.java:215)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.tryAppend(AbstractOutputStreamAppender.java:208)
at org.apache.logging.log4j.core.appender.AbstractOutputStreamAppender.append(AbstractOutputStreamAppender.java:199)
at org.apache.logging.log4j.core.config.AppenderControl.tryCallAppender(AppenderControl.java:161)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender0(AppenderControl.java:134)
at org.apache.logging.log4j.core.config.AppenderControl.callAppenderPreventRecursion(AppenderControl.java:125)
at org.apache.logging.log4j.core.config.AppenderControl.callAppender(AppenderControl.java:89)
at org.apache.logging.log4j.core.config.LoggerConfig.callAppenders(LoggerConfig.java:675)
at org.apache.logging.log4j.core.config.LoggerConfig.processLogEvent(LoggerConfig.java:633)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:616)
at org.apache.logging.log4j.core.config.LoggerConfig.log(LoggerConfig.java:552)
at org.apache.logging.log4j.core.config.AwaitCompletionReliabilityStrategy.log(AwaitCompletionReliabilityStrategy.java:82)
at org.apache.logging.log4j.core.Logger.log(Logger.java:161)
at org.apache.logging.log4j.spi.AbstractLogger.tryLogMessage(AbstractLogger.java:2205)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageTrackRecursion(AbstractLogger.java:2159)
at org.apache.logging.log4j.spi.AbstractLogger.logMessageSafely(AbstractLogger.java:2142)
at org.apache.logging.log4j.spi.AbstractLogger.logMessage(AbstractLogger.java:2017)
at org.apache.logging.log4j.spi.AbstractLogger.logIfEnabled(AbstractLogger.java:1983)
at org.apache.logging.slf4j.Log4jLogger.debug(Log4jLogger.java:139)
at org.apache.spark.internal.Logging.logDebug(Logging.scala:82)
at org.apache.spark.internal.Logging.logDebug$(Logging.scala:81)
at org.apache.spark.rpc.netty.NettyRpcEnv.logDebug(NettyRpcEnv.scala:45)
at org.apache.spark.rpc.netty.NettyRpcEnv$FileDownloadCallback.onFailure(NettyRpcEnv.scala:454)
at org.apache.spark.network.client.TransportResponseHandler.handle(TransportResponseHandler.java:260)

while the class loading lock 0x00000003c0753f88 is locked by ExecutorClassLoader who‘s downloading remote classes/jars though it.

Issue Links

• links to: [Github] Pull Request #35765 (yaooqinn)