[FLINK-6849] Refactor operator state backend and internal operator state hierarchy

• Type: Improvement
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Runtime / State Backends
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Tzu-Li (Gordon) Tai
• Created: 05/Jun/17 10:07
• Updated: 17/Dec/21 22:38

Description

Currently, compared to the keyed state backends, the operator state backends, as well as operator state interfaces, lacks proper hierarchy.

One issue with this lack of hierarchy is that the general concerns of implementing state registration is different between the keyed and operator backends (aside from what is naturally different, such as namespace and key which is not relevant for the operator backend). For example, in the keyed backend hierarchy, AbstractKeyedStateBackend has caches that shortcuts re-accessing already registered state. This behaviour is missing in the operator backend hierarchy, and for example needs to be explicitly handled by the concrete DefaultOperatorStateBackend subclass implementation.

As of now, the need of a proper hierarchy also on the operator backend side might not be that prominent, but will mostly likely become more prominent as we wish to introduce more state structures for operator state (e.g. a MapState for operator state has already been discussed a few times already) as well as more options besides memory-backed operator state.