Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-24417 Build and Run Spark on JDK11
  3. SPARK-28110

on JDK11, IsolatedClientLoader must be able to load java.sql classes

    XMLWordPrintableJSON

    Details

    • Type: Sub-task
    • Status: Closed
    • Priority: Major
    • Resolution: Duplicate
    • Affects Version/s: 3.0.0
    • Fix Version/s: None
    • Component/s: SQL
    • Labels:
      None

      Description

      This might be very specific to my fork & a kind of weird system setup I'm working on, I haven't completely confirmed yet, but I wanted to report it anyway in case anybody else sees this.

      When I try to do anything which touches the metastore on java11, I immediately get errors from IsolatedClientLoader that it can't load anything in java.sql. eg.

      scala> spark.sql("show tables").show()
      java.lang.ClassNotFoundException: java.lang.NoClassDefFoundError: java/sql/SQLTransientException when creating Hive client using classpath: file:/home/systest/jdk-11.0.2/, ...
      ...
      Caused by: java.lang.ClassNotFoundException: java.sql.SQLTransientException
        at java.base/java.net.URLClassLoader.findClass(URLClassLoader.java:471)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:588)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.doLoadClass(IsolatedClientLoader.scala:230)
        at org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1.loadClass(IsolatedClientLoader.scala:219)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
      

      After a bit of debugging, I also discovered that the rootClassLoader is null in IsolatedClientLoader. I think this would work if either rootClassLoader could load those classes, or if isShared() was changed to allow any class starting with "java." (I'm not sure why it only allows "java.lang" and "java.net" currently.)

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                Unassigned
                Reporter:
                irashid Imran Rashid
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: