Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5991

Expose Broadcast Operator State through public APIs



      The broadcast operator state functionality was added in FLINK-5265, it just hasn't been exposed through any public APIs yet.

      Currently, we have 2 streaming connector features for 1.3 that are pending on broadcast state: rescalable Kinesis / Kafka consumers with shard / partition discovery (FLINK-4821 & FLINK-4022). We should consider exposing broadcast state for the 1.3 release also.

      This JIRA also serves the purpose to discuss how we want to expose it.

      To initiate the discussion, I propose:

      1. For the more powerful CheckpointedFunction, add the following to the OperatorStateStore interface:

      <S> ListState<S> getBroadcastOperatorState(ListStateDescriptor<S> stateDescriptor);
      <T extends Serializable> ListState<T> getBroadcastSerializableListState(String stateName);

      2. For a simpler ListCheckpointed variant, we probably should have a separate BroadcastListCheckpointed interface.

      Extending ListCheckpointed to let the user define either the list state type of either PARTITIONABLE or BROADCAST might also be possible, if we can rely on a contract that the value doesn't change. Or we expose a defining method (e.g. getListStateType()) that is called only once in the operator. This would break user code, but can be considered because it is marked as PublicEvolving.


          Issue Links



              • Assignee:
                tzulitai Tzu-Li (Gordon) Tai
                tzulitai Tzu-Li (Gordon) Tai
              • Votes:
                0 Vote for this issue
                4 Start watching this issue


                • Created: