Details
-
New Feature
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.5.0
-
None
Description
A current limitation of Scala UDFs is that a Spark cluster would only be able to support a single REPL at a time due to the fact that classloaders of different Spark Sessions (and therefore, Spark Connect sessions) aren't isolated from each other. Without isolation, REPL-generated classfiles as well as user-added JARs may conflict if there are multiple users of the cluster.
Thus, we need a mechanism to support isolated sessions (i.e isolated resources/classloader) so that each REPL user does not conflict with other users on the same cluster.