When deserializing tasks on executors, we sometimes see IOException: unexpected exception type:
Here are some occurrences of this bug reported on the mailing list and GitHub:
This is probably caused by throwing exceptions other than IOException from our custom readExternal methods (see http://grepcode.com/file/repository.grepcode.com/java/root/jdk/openjdk/7u40-b43/java/io/ObjectStreamClass.java#1022). Davies Liu spotted an instance of this in TorrentBroadcast, where a failed require throws a different exception, but this issue has been reported in Spark 1.1.0 as well. To fix this, I'm going to add try-catch blocks around all of our readExternal and writeExternal methods to re-throw caught exceptions as IOException.
This fix should allow us to determine the actual exceptions that are causing deserialization failures.