[FLINK-3514] Add support for slowly changing streaming broadcast variables

• Type: New Feature
• Status: Open
• Priority: Not a Priority
• Resolution: Unresolved
• Component/s: API / DataStream
• Labels: auto-deprioritized-major auto-deprioritized-minor
• Assignee: Unassigned
• Reporter: Stephan Ewen
• Created: 25/Feb/16 17:58
• Updated: 08/Feb/22 22:37

Description

With "slow changing inputs", I mean auxiliary data sets that change over time. Think about dictionaries, set of rules, etc that are updates at slow rates and then applied onto another stream.

They are a bit like broadcast variables in the DataSet API, but not constant over the entire lifetime of the job.

One can currently approximate this via connected streams:

DataStream<Rule> rules = ...
DataStream<Event> events = ...

rules.broadcast().connect(events).flatMap(...);

The disadvantage is that one has no control that the broadcasted "rules" input is materialized in the function before the "events" are delivered.

Issue Links

• relates to: FLINK-2320 Enable DataSet DataStream Joins