[SPARK-37556] Deser void class fail with Java serialization

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 3.2.0
• Fix Version/s: 3.1.3, 3.0.4, 3.2.1, 3.3.0
• Component/s: Spark Core
• Target Version/s: 3.3.0
• Assignee: Daniel Dai
• Reporter: Daniel Dai
• Created: 06/Dec/21 06:40
• Updated: 07/Dec/21 15:31
• Resolved: 07/Dec/21 15:31

Description

Spark code contains Java void type cannot be serialized with JavaSerializer. For example:

class Foo extends Serializable {
  Class k = Void.TYPE;
}

Spark will throw error:

java.lang.Void; local class name incompatible with stream class name "void"
	at java.io.ObjectStreamClass.initNonProxy(ObjectStreamClass.java:703)
	at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1940)
	at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1806)
	at java.io.ObjectInputStream.readClass(ObjectInputStream.java:1771)
        ......

All other primitive type works. The reason is when SPARK-8730 try to fix the primitive type deserialization, it introduces the following code:

private object JavaDeserializationStream {
  val primitiveMappings = Map[String, Class[_]](
    "boolean" -> classOf[Boolean],
    "byte" -> classOf[Byte],
    "char" -> classOf[Char],
    "short" -> classOf[Short],
    "int" -> classOf[Int],
    "long" -> classOf[Long],
    "float" -> classOf[Float],
    "double" -> classOf[Double],
    "void" -> classOf[Void]
  )
}

However, classOf[Void] is not the equivalence of other types. It's point to Void.class not Void.Type. The equivalence for void should be classOf[Unit]:

scala> classOf[Long]
val res0: Class[Long] = long

scala> classOf[Double]
val res1: Class[Double] = double

scala> classOf[Byte]
val res2: Class[Byte] = byte

scala> classOf[Void]
val res3: Class[Void] = class java.lang.Void  <--- this is wrong

scala> classOf[Unit]
val res4: Class[Unit] = void <---- this is right

Issue Links

• links to: [Github] Pull Request #34816 (daijyc)
• links to: [Github] Pull Request #34816 (daijyc)