[FLINK-22389] Expose Message Batches to SDKs

• Type: New Feature
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: Stateful Functions
• Labels: auto-deprioritized-major auto-deprioritized-minor auto-unassigned
• Assignee: Unassigned
• Reporter: Konstantin Knauf
• Created: 21/Apr/21 07:39
• Updated: 14/Jul/22 22:38

Description

Stateful Functions always invokes a functions with a batch of messages. Under normal operations the batch size=1 (no batching).

If a function is slow and backpressure arises, the batch size grows, though. From an SDK perspective, this batching is not visible. The function is always invoked with a single message. This makes it impossible to efficiently evaluate the whole batch at once (e.g. with pandas).

This issue was requested in http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Statefun-Python-Batch-tp43022.html.