As per discussion on the mailing list
- We add a dedicated streaming mode for now. The streaming mode supersedes the batch mode, so it can run both type of programs.
- The streaming mode sets the memory manager to "lazy allocation".
-> So long as it runs pure streaming jobs, the full heap will be
available to window buffers and UDFs.
-> Batch programs can still run, so mixed workloads are not prevented.
Batch programs are a bit less robust there, because the memory manager does not pre-allocate memory. UDFs can eat into Flink's memory portion.
- The streaming mode starts the necessary configured components/services for state backups
Over the next versions, we want to bring these things together:
- use the managed memory for window buffers
- on-demand starting of the state backend
Then, we deprecate the streaming mode, let both modes start the cluster in the same way.