[FLINK-22741] Hide Flink complexity from Stateful Functions

• Type: New Feature
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Affects Version/s: statefun-3.0.0
• Component/s: Stateful Functions
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Stephan Ewen
• Created: 21/May/21 08:22
• Updated: 03/Aug/22 22:38

Description

This is an umbrella issue for various issues to hide and reduce the complexity and surface area (and configuration space) of Apache Flink when using Stateful Functions.

The goal of this is to create a setup and configuration that works robustly in the vast majority of settings. Users should not be required to configure anything Flink-specific, but only general parameters (like for example total memory size) and StateFun specific parameters (like request timeouts and batching).

If this happens at the cost of some minor regression in peak stream throughput, we can most likely stomach that in StateFun, because the performance cost is commonly dominated by the interaction between StateFun cluster and remote function deployments.

Sub-Tasks

• Reduce JVM Metaspace memory for StateFun: Open
• Add robust default state configuration to StateFun: Closed
• Activate log-based Checkpoints for StateFun: Open