Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6178 Allow upgrades to state serializers
  3. FLINK-6425

Integrate serializer reconfiguration into state restore flow to activate serializer upgrades

    Details

      Description

      With FLINK-6191, TypeSerializer will be reconfigurable.

      From the state backends' point of view, serializer reconfiguration doubles as a mechanism to determine how serializer upgrades should be handled.

      The general idea is that state checkpoints should contain the following as the state's metainfo:

      • the previous serializer
      • snapshot of the previous serializer's configuration

      The upgrade flow is as follows:
      1. On restore, try to deserialize the previous old serializer. Deserialization may fail if a) the serializer no longer exists in classpath, or b) the serializer class is not longer valid (i.e., implementation changed and resulted in different serialVersionUID). In this case, use a dummy serializer as a placeholder. This dummy serializer is currently the ClassNotFoundProxySerializer in the code.

      2. Deserialize the configuration snapshot of the previous old serializer. The configuration snapshot must be successfully deserialized, otherwise the state restore fails.

      3. When we get the new registered serializer for the state (could be a completely new serializer, the same serializer with different implementations, or the exact same serializer untouched; either way they are seen as a new serializer), we use the configuration snapshot of the old serializer to reconfigure the new serializer.

      This completes the upgrade of the old serializer. However, depending on the result of the upgrade, state conversion needs to take place (for now, if state conversion is required, we just fail the job as this functionality isn't available yet). The results could be:

      • Compatible: restore success + serializer upgraded.
      • Compatible, but serialization schema changed: serializer upgraded but requires state conversion, without the requirement that the old serializer needs to be present.
      • Incompatible: serializer upgraded requires state conversion, but requires the old serializer to be present (i.e., can not be the dummy ClassNotFoundProxySerializer).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tzulitai Tzu-Li (Gordon) Tai
                Reporter:
                tzulitai Tzu-Li (Gordon) Tai
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: