Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-2084

Create a dedicated streaming mode

    XMLWordPrintableJSON

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Implemented
    • Affects Version/s: 0.9
    • Fix Version/s: 0.9
    • Component/s: Runtime / Coordination
    • Labels:
      None

      Description

      As per discussion on the mailing list

      http://mail-archives.apache.org/mod_mbox/flink-dev/201505.mbox/browser

      • We add a dedicated streaming mode for now. The streaming mode supersedes the batch mode, so it can run both type of programs.
      • The streaming mode sets the memory manager to "lazy allocation".
        -> So long as it runs pure streaming jobs, the full heap will be
        available to window buffers and UDFs.
        -> Batch programs can still run, so mixed workloads are not prevented.
        Batch programs are a bit less robust there, because the memory manager does not pre-allocate memory. UDFs can eat into Flink's memory portion.
      • The streaming mode starts the necessary configured components/services for state backups

      Over the next versions, we want to bring these things together:

      • use the managed memory for window buffers
      • on-demand starting of the state backend

      Then, we deprecate the streaming mode, let both modes start the cluster in the same way.

        Attachments

          Activity

            People

            • Assignee:
              sewen Stephan Ewen
              Reporter:
              sewen Stephan Ewen
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: