Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-22389

Expose Message Batches to SDKs

Attach filesAttach ScreenshotAdd voteVotersWatch issueWatchersCreate sub-taskLinkCloneUpdate Comment AuthorReplace String in CommentUpdate Comment VisibilityDelete Comments
    XMLWordPrintableJSON

    Details

      Description

      Stateful Functions always invokes a functions with a batch of messages. Under normal operations the batch size=1 (no batching).

      If a function is slow and backpressure arises, the batch size grows, though. From an SDK perspective, this batching is not visible. The function is always invoked with a single message. This makes it impossible to efficiently evaluate the whole batch at once (e.g. with pandas).

      This issue was requested in http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Flink-Statefun-Python-Batch-tp43022.html.

        Attachments

          Activity

            People

            • Assignee:
              Unassigned
              Reporter:
              knaufk Konstantin Knauf

              Dates

              • Created:
                Updated:

                Issue deployment