[SPARK-21569] Internal Spark class needs to be kryo-registered

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Incomplete
• Affects Version/s: 2.2.0
• Component/s: Spark Core
• Labels: bulk-closed
• Assignee: Unassigned
• Reporter: Ryan Williams
• Created: 29/Jul/17 00:07
• Updated: 10/Feb/22 01:10
• Resolved: 25/May/21 01:41

Description

Full repro here

As of 2.2.0, saveAsNewAPIHadoopFile jobs fail (when spark.kryo.registrationRequired=true) with:

java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage
Note: To register this class use: kryo.register(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage.class);
	at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:458)
	at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:79)
	at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:488)
	at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:593)
	at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:315)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:383)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)

This internal Spark class should be kryo-registered by Spark by default.

This was not a problem in 2.1.1.

Issue Links

• blocks: KYLIN-3272 Upgrade Spark dependency to 2.3.2
• duplicates: SPARK-25100 Using KryoSerializer and setting registrationRequired true can lead job failed