[SPARK-1480] Choose classloader consistently inside of Spark codebase

• Type: Improvement
• Status: Resolved
• Priority: Blocker
• Resolution: Fixed
• Fix Version/s: 1.0.0
• Component/s: Spark Core
• Assignee: Patrick Wendell
• Reporter: Patrick Wendell
• Created: 13/Apr/14 03:50
• Updated: 23/Mar/15 08:38
• Resolved: 14/Apr/14 17:14

Description

The Spark codebase is not always consistent on which class loader it uses when classlaoders are explicitly passed to things like serializers. This caused SPARK-1403 and also causes a bug where when the driver has a modified context class loader it is not translated correctly in local mode to the (local) executor.

In most cases what we want is the following behavior:
1. If there is a context classloader on the thread, use that.
2. Otherwise use the classloader that loaded Spark.

We should just have a utility function for this and call that function whenever we need to get a classloader.

Note that SPARK-1403 is a workaround for this exact problem (it sets the context class loader because downstream code assumes it is set). Once this gets fixed in a more general way SPARK-1403 can be reverted.

Issue Links

• is duplicated by: SPARK-1435 Don't assume context class loader is set when creating classes via reflection
• is related to: SPARK-1403 Spark on Mesos does not set Thread's context class loader