Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5089

Introduce Loop functions and Enforce Nesting on Data Streams

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Open
    • Priority: Major
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: None
    • Component/s: API / DataStream
    • Labels:
      None

      Description

      The current programming model of stream iterations in the DataStream API is exposed through low-level dataflow operations for placing arbitrary feedback-edges. This strategy does not enforce proper nesting and requires user-defined timeouts to enforce termination. It further impacts correctness, resulting to incomplete processing or/and inconsistent state when iterations exist in the topology. There is a need for a new functional, yet compositional API (i.e. nested loops) for expressing asynchronous DataStream loops to be able to build easily decentralized progress establishment algorithms such as distributed termination and parallel structured or fixpoint iterations.

      This Jira explains parts of the needs addressed by FLIP-15 on Loops and Terminations.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                senorcarbone Paris Carbone
                Reporter:
                senorcarbone Paris Carbone
              • Votes:
                5 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated: