Details
-
Improvement
-
Status: Resolved
-
P3
-
Resolution: Fixed
-
None
-
None
Description
We should make it easier to configure a Flink state backend. At the moment, users have to either (1) configure the default state backend in their Flink cluster, or make sure (2a) they include the dependency in their Gradle/Maven project (e.g. "org.apache.flink:flink-statebackend-rocksdb_2.11:$flink_version" for RocksDB) (2b) set the state backend factory in the {{FlinkPipelineOptions}.
The drawback of option (2) is that it only works in Java due to the factory specification being in Java.
We can make it easier by simple adding pipeline options for the state backend name and the checkpoint directory which will be enough for configuring the state backend. We can add the RocksDB state backend as a default dependency.
