Details
-
Bug
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
1.6.4, 1.7.2, 1.8.0
-
None
Description
Currently, the KryoSerializerSnapshot only covers information about registered types / serializers that was configured in the ExecutionConfig.
This is problematic, because there are a few cases where we have some additional registrations:
1) When Avro is present in the classpath [1] [2]
2) When Scala is used, in which case Twitter Chill is used which itself has some registrations [3]
3) If a non-registered type is encountered, Kryo will on-the-fly registered the type because we currently configure Kryo to allow dynamic registrations [4].
For case 1), we do reflect these additional registrations in the KryoSerializerSnapshot.
This isn't the case for 2) and 3), which would cause problems when attempting to create a reconfigured instance of the KryoSerializer on restore.
In general, instead of relying on trying to keep track of the registrations ourselves, it would be much more straightforward if there is a way to "dump" the state / configuration of Kryo when we attempt to create a snapshot of the KryoSerializer.
Whether or not Kryo has APIs to allow this needs further investigation.
[1] https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/utils/AvroKryoSerializerUtils.java#L51
[2] https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/utils/AvroKryoSerializerUtils.java#L68
[3] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java#L430
[4] https://github.com/apache/flink/blob/master/flink-core/src/main/java/org/apache/flink/api/java/typeutils/runtime/kryo/KryoSerializer.java#L476