[SPARK-25100] Using KryoSerializer and setting registrationRequired true can lead job failed

• Type: Bug
• Status: Resolved
• Priority: Major
• Resolution: Fixed
• Affects Version/s: 2.3.1
• Fix Version/s: 3.0.0
• Component/s: Spark Core
• Assignee: Deshan Xiao
• Reporter: Deshan Xiao
• Created: 13/Aug/18 10:37
• Updated: 10/Feb/22 01:10
• Resolved: 15/Dec/19 01:15

Description

When spark.serializer is `org.apache.spark.serializer.KryoSerializer` and  `spark.kryo.registrationRequired` is true in SparkConf. I invoked  saveAsNewAPIHadoopDataset to store data in hdfs. The job will fail because the class TaskCommitMessage hasn't be registered.

 

java.lang.IllegalArgumentException: Class is not registered: org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage
Note: To register this class use: kryo.register(org.apache.spark.internal.io.FileCommitProtocol$TaskCommitMessage.class);
at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488)
at com.twitter.chill.KryoBase.getRegistration(KryoBase.scala:52)
at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97)
at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517)
at com.esotericsoftware.kryo.Kryo.writeClassAndObject(Kryo.java:622)
at org.apache.spark.serializer.KryoSerializerInstance.serialize(KryoSerializer.scala:347)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:393)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

 

Issue Links

• is duplicated by: SPARK-21569 Internal Spark class needs to be kryo-registered
• links to: [Github] Pull Request #22093 (deshanxiao)
• links to: GitHub Pull Request #22093
• links to: GitHub Pull Request #26714
• links to: GitHub Pull Request #26916