[FLINK-2084] Create a dedicated streaming mode

• Type: Improvement
• Status: Closed
• Priority: Major
• Resolution: Implemented
• Affects Version/s: 0.9
• Fix Version/s: 0.9
• Component/s: Runtime / Coordination
• Assignee: Stephan Ewen
• Reporter: Stephan Ewen
• Created: 22/May/15 15:22
• Updated: 26/May/15 14:24
• Resolved: 26/May/15 14:24

Description

As per discussion on the mailing list

http://mail-archives.apache.org/mod_mbox/flink-dev/201505.mbox/browser

• We add a dedicated streaming mode for now. The streaming mode supersedes the batch mode, so it can run both type of programs.

• The streaming mode sets the memory manager to "lazy allocation".
  -> So long as it runs pure streaming jobs, the full heap will be
  available to window buffers and UDFs.
  -> Batch programs can still run, so mixed workloads are not prevented.
  Batch programs are a bit less robust there, because the memory manager does not pre-allocate memory. UDFs can eat into Flink's memory portion.

• The streaming mode starts the necessary configured components/services for state backups

Over the next versions, we want to bring these things together:

• use the managed memory for window buffers
• on-demand starting of the state backend

Then, we deprecate the streaming mode, let both modes start the cluster in the same way.

Sub-Tasks

• Add an option to the MemoryManager to allocate memory as needed, rather than preallocating it: Closed
• Add streaming mode to YARN as well: Resolved