[SPARK-28110] on JDK11, IsolatedClientLoader must be able to load java.sql classes

• Type: Sub-task
• Status: Closed
• Priority: Major
• Resolution: Duplicate
• Affects Version/s: 3.0.0
• Component/s: SQL
• Assignee: Unassigned
• Reporter: Imran Rashid
• Created: 19/Jun/19 19:03
• Updated: 16/Aug/19 09:30
• Resolved: 14/Aug/19 22:34

Description

This might be very specific to my fork & a kind of weird system setup I'm working on, I haven't completely confirmed yet, but I wanted to report it anyway in case anybody else sees this.

When I try to do anything which touches the metastore on java11, I immediately get errors from IsolatedClientLoader that it can't load anything in java.sql. eg.

scala> spark.sql("show tables").show()
java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: java/sql/SQLTransientException when creating Hive client using classpath: file:/home/systest/jdk-11.0.2/, ...
...
Caused by: java.lang.ClassNotFoundException: java.sql.SQLTransientException
  at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:230)
  at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:219)
  at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)

After a bit of debugging, I also discovered that the rootClassLoader is null in IsolatedClientLoader. I think this would work if either rootClassLoader could load those classes, or if isShared() was changed to allow any class starting with "java." (I'm not sure why it only allows "java.lang" and "java.net" currently.)

Issue Links

• is a clone of: SPARK-26839 Work around classloader changes in Java 9 for Hive isolation
• is superceded by: SPARK-28723 Upgrade to Hive 2.3.6 for HiveMetastore Client and Hadoop-3.2 profile