Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-6804

Inconsistent state migration behaviour between different state backends

    Details

      Description

      The MemoryStateBackend, FsStateBackend and RocksDBStateBackend show a different behaviour when it comes to recovery from old state and state migration. For example, using the MemoryStateBackend it is possible to recover pojos which now have additional fields (at recovery time). The only caveat is that the recovered PojoSerializer will silently drop the added fields when writing the new Pojo. In contrast, the RocksDBStateBackend correctly recognizes that a state migration is necessary and thus fails with a StateMigrationException. The same applies to the case where Pojo field types change. The MemoryStateBackend and the FsStateBackend accept such a change as long as the fields still have the same length. The RocksDBStateBackend correctly fails with a StateMigrationException.

      I think that all state backends should behave similarly and give the user the same recovery and state migration guarantees. Otherwise, it could happen that jobs run with one state backend but not with another (wrt semantic behaviour).

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tzulitai Tzu-Li (Gordon) Tai
                Reporter:
                till.rohrmann Till Rohrmann
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: