[FLINK-11947] Support MapState value schema evolution for RocksDB

• Type: Improvement
• Status: Resolved
• Priority: Critical
• Resolution: Fixed
• Fix Version/s: 1.9.0
• Component/s: API / Type Serialization System, Runtime / State Backends
• Labels: pull-request-available
• Assignee: Congxian Qiu
• Reporter: Tzu-Li (Gordon) Tai
• Created: 18/Mar/19 05:05
• Updated: 20/Jun/19 02:50
• Resolved: 20/Jun/19 02:50

Description

Currently, we do not attempt to perform state schema evolution if the key or value's schema of a user MapState has changed when using RocksDB:

https://github.com/apache/flink/blob/953a5ffcbdae4115f7d525f310723cf8770779df/flink-state-backends/flink-statebackend-rocksdb/src/main/java/org/apache/flink/contrib/streaming/state/RocksDBKeyedStateBackend.java#L542

This was disallowed in the initial support for state schema evolution because the way we did state evolution in the RocksDB state backend was simply overwriting values.

For MapState key evolution, only overwriting RocksDB values does not work, since RocksDB entries for MapState uses a composite key containing the map state key. This means that when evolving MapState in this case with an evolved key schema, we will have new entries.

Issue Links

• links to: GitHub Pull Request #8565