Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-44078

Add support for classloader/resource isolation

    XMLWordPrintableJSON

Details

    • New Feature
    • Status: Resolved
    • Major
    • Resolution: Fixed
    • 3.5.0
    • 3.5.0
    • Connect, Spark Core
    • None

    Description

      A current limitation of Scala UDFs is that a Spark cluster would only be able to support a single REPL at a time due to the fact that classloaders of different Spark Sessions (and therefore, Spark Connect sessions) aren't isolated from each other. Without isolation, REPL-generated classfiles as well as user-added JARs may conflict if there are multiple users of the cluster.

      Thus, we need a mechanism to support isolated sessions (i.e isolated resources/classloader) so that each REPL user does not conflict with other users on the same cluster.

      Attachments

        Activity

          People

            vicennial Venkata Sai Akhil Gudesa
            vicennial Venkata Sai Akhil Gudesa
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: