Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-24858

TypeSerializer version mismatch during eagerly restore

    XMLWordPrintableJSON

Details

    • Hide
      This ticket resolves an issue that during state migration between Flink versions the wrong serializer might have been picked.

      When upgrading from Flink 1.13.x please immediately choose 1.14.3 or higher and skip 1.14.0, 1.14.1, 1.14.2 because all are affected and it might prevent your job from starting.
      Show
      This ticket resolves an issue that during state migration between Flink versions the wrong serializer might have been picked. When upgrading from Flink 1.13.x please immediately choose 1.14.3 or higher and skip 1.14.0, 1.14.1, 1.14.2 because all are affected and it might prevent your job from starting.

    Description

      Currently, some of our TypeSerializer snapshots assume information about the binary layout of the actual data rather than only holding information about the TypeSerialzer.

      Multiple users ran into this problem i.e.https://lists.apache.org/thread/4q5q7wp0br96op6p7f695q2l8lz4wfzx

      This manifest itself when state is restored egarly (for example an operator state) but, for example a user doesn't register the state on their intializeState/open,* and then a checkpoint happens.
      The result is that we will have elements serialized according to an old binary layout, but our serializer snapshot declares a new version which indicates that the elements are written with a new binary layout.
      The next restore will fail.

      Attachments

        Activity

          People

            fpaul Fabian Paul
            fpaul Fabian Paul
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: