[SPARK-20547] ExecutorClassLoader's findClass may not work correctly when a task is cancelled.

• Type: Bug
• Status: Resolved
• Priority: Minor
• Resolution: Fixed
• Affects Version/s: 2.1.0, 2.2.0
• Fix Version/s: 3.0.0
• Component/s: Spark Shell
• Assignee: Shixiong Zhu
• Reporter: Shixiong Zhu
• Created: 01/May/17 20:57
• Updated: 04/Aug/19 13:42
• Resolved: 28/May/19 19:57

Description

ExecutorClassLoader's findClass may throw some transient exception. For example, when a task is cancelled, if ExecutorClassLoader is running, you may see InterruptedException or IOException, even if this class can be loaded. Then the result of findClass will be cached by JVM, and later when the same class is being loaded (note: in this case, this class may be still loadable), it will just throw NoClassDefFoundError.

We should make ExecutorClassLoader retry on transient exceptions.

Issue Links

• links to: GitHub Pull Request #24683