Uploaded image for project: 'Flink'
  1. Flink
  2. FLINK-5991

Expose Broadcast Operator State through public APIs

    Details

      Description

      The broadcast operator state functionality was added in FLINK-5265, it just hasn't been exposed through any public APIs yet.

      Currently, we have 2 streaming connector features for 1.3 that are pending on broadcast state: rescalable Kinesis / Kafka consumers with shard / partition discovery (FLINK-4821 & FLINK-4022). We should consider exposing broadcast state for the 1.3 release also.

      This JIRA also serves the purpose to discuss how we want to expose it.

      To initiate the discussion, I propose:

      1. For the more powerful CheckpointedFunction, add the following to the OperatorStateStore interface:

      <S> ListState<S> getBroadcastOperatorState(ListStateDescriptor<S> stateDescriptor);
      
      <T extends Serializable> ListState<T> getBroadcastSerializableListState(String stateName);
      

      2. For a simpler ListCheckpointed variant, we probably should have a separate BroadcastListCheckpointed interface.

      Extending ListCheckpointed to let the user define either the list state type of either PARTITIONABLE or BROADCAST might also be possible, if we can rely on a contract that the value doesn't change. Or we expose a defining method (e.g. getListStateType()) that is called only once in the operator. This would break user code, but can be considered because it is marked as PublicEvolving.

        Attachments

          Issue Links

            Activity

              People

              • Assignee:
                tzulitai Tzu-Li (Gordon) Tai
                Reporter:
                tzulitai Tzu-Li (Gordon) Tai
              • Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: