[FLINK-24858] TypeSerializer version mismatch during eagerly restore - ASF JIRA

XML

Word

Printable

JSON

Details

Type: Bug
Status: Resolved
Priority: Blocker
Resolution: Fixed
Affects Version/s: 1.14.0, 1.13.3, 1.15.0
Fix Version/s: 1.14.3, 1.15.0
Component/s: API / Type Serialization System
Labels:
- pull-request-available

Release Note:

Hide
This ticket resolves an issue that during state migration between Flink versions the wrong serializer might have been picked.

When upgrading from Flink 1.13.x please immediately choose 1.14.3 or higher and skip 1.14.0, 1.14.1, 1.14.2 because all are affected and it might prevent your job from starting.

Show
This ticket resolves an issue that during state migration between Flink versions the wrong serializer might have been picked. When upgrading from Flink 1.13.x please immediately choose 1.14.3 or higher and skip 1.14.0, 1.14.1, 1.14.2 because all are affected and it might prevent your job from starting.

Description

Currently, some of our TypeSerializer snapshots assume information about the binary layout of the actual data rather than only holding information about the TypeSerialzer.

Multiple users ran into this problem i.e.https://lists.apache.org/thread/4q5q7wp0br96op6p7f695q2l8lz4wfzx

This manifest itself when state is restored egarly (for example an operator state) but, for example a user doesn't register the state on their intializeState/open,* and then a checkpoint happens.
The result is that we will have elements serialized according to an old binary layout, but our serializer snapshot declares a new version which indicates that the elements are written with a new binary layout.
The next restore will fail.