Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Fixed
-
3.1.0-incubating
-
None
Description
SparkGraphComputer.submit delegates most of its work to a closure that executes on the common forkjoin pool. The closure does a lot of stuff. It calls into both Spark and Hadoop.
This approach has two problems:
1. Inability to customize the context classloader used within the closure
The context classloader of the thread that called submit is not necessarily the same as the context classloader common forkjoin pool threads. This matters because multiple bits of code reachable from submit's closure rely on the context classloader. SparkMemory is one; Hadoop's UserGroupInformation is another, depending on the credentials configuration (UGI is reached indirectly via FileSystem.get). This basically means that the caller has to use whatever context classloader is currently in use by the fork join common pool, or else bad things can happen, such as nonsensical-looking ClassCastExceptions.
2. Inability to override the context classloader inside the closure
When System.getSecurityManager() != null, the common forkjoin pool switches from its default worker thread factory implementation to a more restrictive alternative called InnocuousForkJoinWorkerThreadFactory. Threads created by this factory can't call setContextClassLoader. Attempting to do so throws a SecurityException. However, UserGroupInformation.newLoginContext must be able to call setContextClassLoader. It saves the CCL to a variable, does some work, then restores the CCL from a variable. This is impossible if the method throws a SecurityException. So, if a security manager is present in the VM, submit's closure can die in FileSystem.get -> UGI before any useful work even begins.
I set the Affects Version to the version on which I observed it, but it might affect earlier versions too.