[FLINK-18316] Add a dynamic state registration primitive for Stateful Functions

• Type: New Feature
• Status: Closed
• Priority: Major
• Resolution: Fixed
• Fix Version/s: statefun-2.2.0
• Component/s: Stateful Functions
• Labels: pull-request-available
• Assignee: Tzu-Li (Gordon) Tai
• Reporter: Tzu-Li (Gordon) Tai
• Created: 16/Jun/20 04:27
• Updated: 13/Jul/20 04:41
• Resolved: 13/Jul/20 04:41

Description

Currently, using the PersistedValue / PersistedTable / PersistedAppendingBuffer primitives, the user can only eagerly define states prior to function instance activation using the Persisted field annotation.

We propose to add a primitive that allows them to register states dynamically after activation (i.e. during runtime), along the lines of:

public MyStateFn implements StatefulFunction {

    @Persisted
    private final PersistedStateProvider provider = new PersistedStateProvider();

    public MyStateFn() {
        PersistedValue<String> valueState = provider.getValue(...);
    }

    void invoke(Object input) {
        PersistedValue<String> anotherValueState = provider.getValue(...);
    }
}

Note how you can register state during instantiation (in the constructor) and in the invoke method. Both registrations should be picked up by the runtime and bound to Flink state.

This will be useful for a few scenarios:

• Could enable us to get rid of eager state spec definitions in the YAML modules for remote functions in the future.
• Will allow new state to be registered in remote functions, without shutting down the StateFun cluster.
• Moreover, this approach allows us to differentiate which functions have dynamic state and which ones have only eager state, which might be handy in the future in case there is a need to differentiate.

Issue Links

• links to: GitHub Pull Request #125