Details
-
New Feature
-
Status: Open
-
Not a Priority
-
Resolution: Unresolved
-
None
-
None
Description
In Flink 1.9 and before, if we want to custom state backend, we have to supply concrete state backend implementation in StreamExecutionEnvironment.setStateBackend. This implies that we have to depend on that concrete state backend implementation in job graph composition phase since we have to construct this concrete state backend. This may not be appropriate if we separate job graph composition and execution strictly, as job graph composition requires no state backend functions besides customization.
Suppose that we are going to build a declarative streaming platform that building plain json/yaml like textual object, which may contain custom state backend configuration, with attached dependent jars to job graph and submitting generated job graph to Flink, we apparently should avoid ad-hoc state backend construction due to code reusability and indeterministic state backends configured.
So I propose to support state backend customization in job graph composition phase to not depend on concrete state backend implementation.
To achieve this goal, a Configuration field, say StreamExecutionEnvironment.stateBackendConfiguration, should be sufficient.