Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-20264 Zero-downtime / dynamic function upgrades in Stateful Functions
  3. FLINK-20265

Extend invocation protocol to allow functions to indicate incomplete invocation context

    XMLWordPrintableJSON

Details

    Description

      Currently, users declare the states a function will access with a module YAML definition file. The modules are loaded once when starting a StateFun cluster, meaning that the state specifications remain static throughout the cluster's execution lifetime.

      We propose that state specifications should be declared by the function themselves via the language SDKs, instead of being declared in the module YAMLs.

      The state specifications, now living in the functions, can be made discoverable by the StateFun runtime through the invocation request-reply protocol.

      Brief simplified sketch of the extended protocol:

      • StateFun dispatches an invocation request, with states [A, B].
      • Function receives request, but since it requires states [A, B, C, D], it responds with a IncompleteInvocationContext response that indicates state values for [C, D] is missing.
      • StateFun receives this response, and registers new Flink state handles for [C, D].
      • Finally, a new invocation request with the same input messages, but "patched" with new states to contain all values for [A, B, C, D] is resent to the function.

      This JIRA only targets updating the Protobuf messages ToFunction and FromFunction to fulfill the extended protocol, and support handling IncompleteInvocationContext responses in the request dispatcher.

      Updating SDKs should be separate subtask JIRAs.

      Attachments

        Issue Links

          blocks

          Sub-task - The sub-task of the issue FLINK-20268 Update Python SDK to implement new invocation protocol

          • Major - Major loss of function.
          • Closed

          Sub-task - The sub-task of the issue FLINK-20334 Introduce function endpoint path templating in module YAML specifications

          • Major - Major loss of function.
          • Closed
          links to

          Web Link GitHub Pull Request #177

          Activity

            People

              tzulitai Tzu-Li (Gordon) Tai
              tzulitai Tzu-Li (Gordon) Tai
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: