[HADOOP-16159] Deadlock when using FsUrlStreamHandlerFactory

• Type: Bug
• Status: Open
• Priority: Critical
• Resolution: Unresolved
• Affects Version/s: 2.8.5, 3.1.2
• Assignee: Unassigned
• Reporter: Ajith S
• Created: 04/Mar/19 06:01
• Updated: 03/May/22 23:33

Description

URL.setURLStreamHandlerFactory(new FsUrlStreamHandlerFactory())

 This induces the thread lock

Thread 1 : Does load class which will do below

• waiting to lock <0x00000005c0c1e5e0> (a org.apache.hadoop.conf.Configuration)
  at org.apache.hadoop.conf.Configuration.handleDeprecation(Configuration.java:684)
  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1088)
  at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
  at org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
  at java.net.URL.getURLStreamHandler(URL.java:1142)
  at java.net.URL.<init>(URL.java:420)
  at sun.misc.URLClassPath$JarLoader.<init>(URLClassPath.java:812)
  at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1094)
  at sun.misc.URLClassPath$JarLoader$3.run(URLClassPath.java:1091)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1090)
  at sun.misc.URLClassPath$JarLoader.getResource(URLClassPath.java:1050)
  at sun.misc.URLClassPath.getResource(URLClassPath.java:239)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:365)
  at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
  at java.security.AccessController.doPrivileged(Native Method)
  at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
• locked <0x00000005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at java.lang.Class.forName0(Native Method)
  at java.lang.Class.forName(Class.java:348)  

Thread 2 : Create new URL

• waiting to lock <0x00000005b7991168> (a org.apache.spark.util.MutableURLClassLoader)
  at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
  at org.apache.xerces.parsers.ObjectFactory.findProviderClass(Unknown Source)
  at org.apache.xerces.parsers.ObjectFactory.newInstance(Unknown Source)
  at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
  at org.apache.xerces.parsers.ObjectFactory.createObject(Unknown Source)
  at org.apache.xerces.parsers.DOMParser.<init>(Unknown Source)
  at org.apache.xerces.parsers.DOMParser.<init>(Unknown Source)
  at org.apache.xerces.jaxp.DocumentBuilderImpl.<init>(Unknown Source)
  at org.apache.xerces.jaxp.DocumentBuilderFactoryImpl.newDocumentBuilder(Unknown Source)
  at org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:2737)
  at org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:2696)
  at org.apache.hadoop.conf.Configuration.getProps(Configuration.java:2579)
• locked <0x00000005c0c1e5e0> (a org.apache.hadoop.conf.Configuration)
  at org.apache.hadoop.conf.Configuration.get(Configuration.java:1091)
  at org.apache.hadoop.conf.Configuration.getTrimmed(Configuration.java:1145)
  at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2363)
  at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2840)
  at org.apache.hadoop.fs.FsUrlStreamHandlerFactory.createURLStreamHandler(FsUrlStreamHandlerFactory.java:74)
  at java.net.URL.getURLStreamHandler(URL.java:1142)
  at java.net.URL.<init>(URL.java:599)

 

Issue Links

• is fixed by: HADOOP-15331 Fix a race condition causing parsing error of java.io.BufferedInputStream in class org.apache.hadoop.conf.Configuration
• relates to: SPARK-31933 Found another deadlock in Spark Driver
• relates to: SPARK-39094 FsUrlStreamHandlerFactory deadlock with Hadoop 2 profile
• relates to: SPARK-26961 Found Java-level deadlock in Spark Driver